23,121
社区成员
发帖
与我相关
我的任务
分享关于pthread多线程虚拟内存的问题
各位前辈，你们好
我写了一个小程序，代码如下
用top命令看了一下，VIRT一直在增长，增长到1百多g，这是为什么呀
？pthread_join()不是回收子线程内存的吗？
我写了一个小程序，代码如下
#include<pthread.h>
#include<iostream>
#include<cstdlib>
using namespace std;
pthread_mutex_t fmx;
void* func(void *x)
{
int *p=(int*)x;
int **data=(int **)malloc((*p)*sizeof(int*));
for(int i=0;i<*p;i++)
{
data[i]=(int*)malloc((*p)*sizeof(int));
for(int j=0;j<*p;j++)
{
data[i][j]=j;
}
}
for(int i=0;i<*p;i++)
{
for(int j=0;j<*p;j++)
{
data[i][j]=data[i][j]*data[i][j];
}
}
for(int i=0;i<*p;i++)
{
free(data[i]);
data[i]=0;
}
free(data);
data=0;
}
int main()
{
int k=2000;
for(int i=0;i<10;i++)
{
pthread_t threadlist[2000];
for(int j=0;j<2000;j++)
{
pthread_create(&threadlist[i],NULL,func,&k);
}
for(int j=0;j<2000;j++)
{
void *r=0;
pthread_join(threadlist[i],&r);
}
cout<<"******************************************patch "<<i<<" over!"<<endl;
}
}
用top命令看了一下，VIRT一直在增长，增长到1百多g，这是为什么呀
？pthread_join()不是回收子线程内存的吗？
...全文
请发表友善的回复…
发表回复
JDSH0224 2014-08-13
- 打赏
- 举报
linux创建线程是有限制的，你可以查下
jack_leiwin 2014-08-03
- 打赏
- 举报
first of all pthread_xxxx functions will never throw any C++ exceptions. You shouldn't expect to determine the exit status of your program by exceptions.
second, all your threads are joinable, which means the system has to keep certain amount of RAM for each of them, so your pthread_join can then pick up the exit status of threads.
http://www.ibm.com/developerworks/library/l-memory-leaks/
If each thread does take 10MB reservation for the worst scenario, then your 2K threads will be 20GB. They will not be released until the pthread_join is called an each of them. In other words, even you think the thread is terminated, the memory will not be released until pthread_join is called.
Solution: detached thread. See the url above for more information
Third, as someone indicated above, it's related to the implementation of malloc in c library. It probably has a memory pool to back it up. If there is no free node available, the pool will expand itself. I am not surprised the expansion will take twice the space it had.
Fourth, the implementation of operating system, I don't think I am qualified to explain anything for this regard. But I am pretty sure it will contributes to what you had.
So, short answer: try detached thread
recommendation: I respect the fact that you tried to dig into bottom of this. But, is it really worth you time? A year after, if you revist this question, you could feel you time was a waste, well, not to mention the time I spend here (more than 20 minutes so far). You know why? you will probably have no chance at all to write this kind of codes in your career. My lesson taught me well: you are luck if you have a job doing whatever you like to do. But, in reality, most of people don't. You see, if I have a dream job, why I wast my time here with you? No offense, this applies to a lot of people who wander around here... [/quote]
thank you for your recommendation, coding is not my favorate job, I'm just doing my best to get all my work done. Computer science is too difficult to me get it, I just do not know how to learn it in a way that suits to me. thak you again
小笨和漂向北方 2014-08-03
- 打赏
- 举报
btw, my major is not CS either. I never took any cs class at all. I guess we are in the same boat in this context. :)
小笨和漂向北方 2014-08-03
- 打赏
- 举报
Then you should have my respect! I am sure there are "other ways" to get your job done. If CS is not your major, but you don't have some programming needs, you may want to try python, matlab. A lot of researchers/scientists I know feel comfortable with them. Even you have requirements on large scale computations, those platforms have answers already.
Good luck!
小笨和漂向北方 2014-08-01
- 打赏
- 举报
first of all pthread_xxxx functions will never throw any C++ exceptions. You shouldn't expect to determine the exit status of your program by exceptions.
second, all your threads are joinable, which means the system has to keep certain amount of RAM for each of them, so your pthread_join can then pick up the exit status of threads.
http://www.ibm.com/developerworks/library/l-memory-leaks/
If each thread does take 10MB reservation for the worst scenario, then your 2K threads will be 20GB. They will not be released until the pthread_join is called an each of them. In other words, even you think the thread is terminated, the memory will not be released until pthread_join is called.
Solution: detached thread. See the url above for more information
Third, as someone indicated above, it's related to the implementation of malloc in c library. It probably has a memory pool to back it up. If there is no free node available, the pool will expand itself. I am not surprised the expansion will take twice the space it had.
Fourth, the implementation of operating system, I don't think I am qualified to explain anything for this regard. But I am pretty sure it will contributes to what you had.
So, short answer: try detached thread
recommendation: I respect the fact that you tried to dig into bottom of this. But, is it really worth you time? A year after, if you revist this question, you could feel you time was a waste, well, not to mention the time I spend here (more than 20 minutes so far). You know why? you will probably have no chance at all to write this kind of codes in your career. My lesson taught me well: you are luck if you have a job doing whatever you like to do. But, in reality, most of people don't. You see, if I have a dream job, why I wast my time here with you? No offense, this applies to a lot of people who wander around here...
小笨和漂向北方 2014-08-01
- 打赏
- 举报
define exception in the context of Linux c programming please.
you wrote a c++ program but everything I saw in there is C except for the cout...
jack_leiwin 2014-08-01
- 打赏
- 举报
前辈，你说的说线程创建失败或者内存没有malloc成功（其实我这里确实有问题，应该都检查一下是否创建或者分配成功）的话，是不是程序会非正常结束，最终会弹出问题提示的，但是我这里应该就是你说的那样，是所有线程都创建成功，新的id覆盖旧的id，内存全分配成功并且正确free掉了，但就是奇怪的是DATA在增长，VIRT也在增长，莫名其妙。[/quote]
how could you be so sure?
Do you have the prove by tools like valgrind?[/quote]
every time I start the project, it just complete and quit normally without any exception.
小笨和漂向北方 2014-08-01
- 打赏
- 举报
这个也倒是有一定的道理[/quote]
VIRT has nothing to do with the real memory usage. they are just virtual address spaces and will not trigger OOM killer if you don't consume them. RSS matters more to the actual memory usage.
小笨和漂向北方 2014-08-01
- 打赏
- 举报
前辈，你说的说线程创建失败或者内存没有malloc成功（其实我这里确实有问题，应该都检查一下是否创建或者分配成功）的话，是不是程序会非正常结束，最终会弹出问题提示的，但是我这里应该就是你说的那样，是所有线程都创建成功，新的id覆盖旧的id，内存全分配成功并且正确free掉了，但就是奇怪的是DATA在增长，VIRT也在增长，莫名其妙。[/quote]
how could you be so sure?
Do you have the prove by tools like valgrind?
jack_leiwin 2014-08-01
- 打赏
- 举报
这个也倒是有一定的道理
jack_leiwin 2014-08-01
- 打赏
- 举报
前辈，你说的说线程创建失败或者内存没有malloc成功（其实我这里确实有问题，应该都检查一下是否创建或者分配成功）的话，是不是程序会非正常结束，最终会弹出问题提示的，但是我这里应该就是你说的那样，是所有线程都创建成功，新的id覆盖旧的id，内存全分配成功并且正确free掉了，但就是奇怪的是DATA在增长，VIRT也在增长，莫名其妙。
jack_leiwin 2014-08-01
- 打赏
- 举报
define exception in the context of Linux c programming please.
you wrote a c++ program but everything I saw in there is C except for the cout...[/quote]
you mean
try{
}cache{
}
if((err=pthread_create(&threadlist[j],NULL,func,&k))!=0)
{
...
}
布鲁克斯南南 2014-07-08
- 打赏
- 举报
大牛不愧是大牛，真细心！[/quote]
yyysss520 2014-07-08
- 打赏
- 举报
内存库有自己的分配策略，很多情况下VIRT显示的是 程序内存的高峰值。你换个操作系统或者内存库（tcmalloc之类的），现象可能就不一样了。一个程序如果有很多内存碎片，也会算到VIRT中。
buyong 2014-07-07
- 打赏
- 举报
用free命令看一下，说不定都是系统cache
阿呆_ 2014-07-05
- 打赏
- 举报
代码中没有任何的错误检查，所以很难说问题在哪。
你不能保证2000个thread都create成功了。毕竟每个新create的thread默认需要分配2M stack的，谁都无法保证2000个2M是否都能够在当前进程地址空间内分配，如果是32位程序的话光堆栈就差不多占光了地址空间了，更别说线程内那无数个malloc了。
不幸的是main函数中保存threadid的数组并未初始化（比如清零），它是有原始值的，这个原始值是程序运行过程中保存在堆栈上的无用随机数，而一旦某个pthread_create失败后这个对应数组位置上的值并不会修改，因此就成了一个谁也不知道是不是有效的threadid（99.9%是一个无效id, 但不排除你人品爆发它可能是一个有效的系统id），然后你pthread_join它...... 生活是万花筒.
另外据我的经验，即使已经成功create的threads函数中的malloc也不会都成功，那么这就意味着有某些thread是必然会非法内存访问的，这也就是说某些thread中已经成功malloc的内存必然会泄漏，因为非法内存访问必定中断thread运行，所以其后的所有free或者delete代码没有机会执行了。
jack_leiwin 2014-07-05
- 打赏
- 举报
为什么呢？这和下面有什么区别吗？
for(int j=0;j<2000;j++)
{
void *r=0;
pthread_join(threadlist[j],&r);
}
cout<<"******************************************patch "<<i<<" over!"<<endl;
imliuda 2014-06-27
- 打赏
- 举报
每创建一个线程join一次，或者用pthread_detach
jack_leiwin 2014-06-15
- 打赏
- 举报
大牛都到哪儿去了？现在都没有别人回一个
jack_leiwin 2014-06-15
- 打赏
- 举报
大神，我在把我观察到的说一下
运行top时VIRT，和DATA一直在增长，RES增长到30g左右就会清空，然后重新增加，VIRT和DATA一直在增长，这个是什么导致的啊？
这个程序总共需要2000*2000*2000*4个B的数据空间，也就是30g多一点，在i=1的时候，VIRT一下子蹦到二十多g，RES呢只增加到六点多g就开始清空了
当i=2时，RES会增加到30g左右然后开始清空，表示本次运行完了，但是VIRT和DATA一直在增加，就没有清空的现象，这是真么导致的呢？
加载更多回复（4）
