介绍


项目进行压力测试时报错org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多

报错内容


16-Sep-2021 20:54:23.275 严重 [http-nio-8064-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:455)
at java.lang.Thread.run(Thread.java:748)

16-Sep-2021 20:56:22.156 严重 [http-nio-8064-exec-32] org.apache.coyote.http11.Http11Processor.service Error processing request
java.lang.IllegalStateException: java.io.FileNotFoundException: /data/tomcat-option/webapps/carsoa-opinion/WEB-INF/lib/swagger-bootstrap-ui-1.9.5.jar (打开的文件过多)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:100)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:262)
at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
at org.apache.catalina.webresources.CachedResource.validateResource(CachedResource.java:110)
at org.apache.catalina.webresources.Cache.getResource(Cache.java:69)
at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216)
at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:206)
at org.apache.catalina.mapper.Mapper.internalMapWrapper(Mapper.java:1027)
at org.apache.catalina.mapper.Mapper.internalMap(Mapper.java:842)
at org.apache.catalina.mapper.Mapper.map(Mapper.java:698)
at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:679)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /data/tomcat-option/webapps/carsoa-opinion/WEB-INF/lib/swagger-bootstrap-ui-1.9.5.jar (打开的文件过多)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:188)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:173)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:313)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:96)
… 20 more

分析1 调整linux系统打开文件数


根据org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多
我们初步判断为系统没有优化
下面我们优化一下系统参数

ulimit -a 查看系统文件打开数为1024,这个明显太小了

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128531
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 128531
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

查看报错程序所占用的文件数

lsof -p 17405wc -l
1005

调整系统文件打开数

vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

重新登录用户生效
再次压测,发现还是有问题

分析2 调整tomcat配置


根据 org.apache.coyote.http11.Http11Processor.service Error processing request
java.lang.IllegalStateException: java.io.FileNotFoundException: /data/tomcat-option/webapps/carsoa-opinion/WEB-INF/lib/swagger-bootstrap-ui-1.9.5.jar (打开的文件过多)
报错,我们发现有可能是tomcat线程配置应该有问题,文件过多
调整tomcat,在server.xml中修改如下一段

<Connector port="8064" protocol="HTTP/1.1"
           connectionTimeout="20000"
           acceptCount="100"
           redirectPort="8443" />

改成

<Connector port="8064" protocol="HTTP/1.1"
           connectionTimeout="20000"
           acceptCount="1000"
           redirectPort="8443" />

调整完成
acceptCount=“100” 改为 acceptCount=“1000”
压测,项目正常响应