java 循环代码优化

[干饭人之路]

		List<MyIssue> AL=new ArrayList<MyIssue>();
		List<MyIssue> issuesAll = new ArrayList();
		//此次略去N行代码给issuesAll和issuesAll赋值
		issuesAll = IssueDao.fromResultSet(resultAll);
		AL= xxxxx;
		
		 //issuesAll.size()=154345
		 //AL.size()=35678434 
		 //下面的两层嵌套循环总次数太多，请问各位有没有办法提高效率？？？？
		 //由于AL.size()远远大于issuesAll.size(),能否用多个进程并行？？？？代码怎么写？？
				for(int j=0;j<issuesAll.size();j++){										
						 for(int i=0;i<AL.size();i++){							
							if(mycontains(AL.get(i).getRedBalls(),issuesAll.get(j).getRedBalls(),a_numlimit)){								
									AL.remove(AL.get(i));								
								}				
						  }				 
					 }

[qq_39936465]

对MyIssue设立一个关键字，2个list然后按关键字排序，循环时可以比较关键字位置，超过位置后面的循环可以直接跳出

[干饭人之路]

最新优化后的代码：

List<MyIssue> AL=new ArrayList<MyIssue>();
List<MyIssue> issuesAll = new ArrayList();
//此次略去N行代码给issuesAll和issuesAll赋值
issuesAll = IssueDao.fromResultSet(resultAll);
AL= xxxxx;
 
 //issuesAll.size()=154345
 //AL.size()=35678434 
 //下面的两层嵌套循环总次数太多，请问各位有没有办法提高效率？？？？
 //由于AL.size()远远大于issuesAll.size(),能否用多个进程并行？？？？代码怎么写？？
       /*
        for(int j=0;j<issuesAll.size();j++){                                       
                 for(int i=0;i<AL.size();i++){                           
                    if(mycontains(AL.get(i).getRedBalls(),issuesAll.get(j).getRedBalls(),a_numlimit)){                               
                            AL.remove(AL.get(i));                               
                        }               
                  }                 
             }
        */

           List<SsqIssue> AL2=new ArrayList<SsqIssue>();
			
          //log("dealListWithMutiThread="+dealListWithMutiThread(AL,issuesAll,a_numlimit).size());
			
	 AL2.addAll(dealListWithMutiThread(AL,issuesAll,a_numlimit));
	 AL.clear();
	 AL.addAll(AL2);
	 AL2.clear();

//多线程处理AL
public static List<SsqIssue> dealListWithMutiThread(List<SsqIssue> AL,List<SsqIssue> issuesAll,int a_numlimit){
    //List<SsqIssue> list = new ArrayList<SsqIssue>(10000);
    
    int index = 0;
    int dealSize = 500000;
    int b=AL.size()/dealSize+1;
    ExecutorService ex = Executors.newFixedThreadPool(b);
    
   
    List<Future<List<SsqIssue>>> futures = new ArrayList<>();
    //log(" AL.size()/dealSize="+ AL.size()/dealSize);
    //sleep(10);
            //分配
    for(int i=0;i<= b;i++,index+=dealSize){
        int start = index;
        if(start>=AL.size()) break;
        int end = start + dealSize;
        end = end>AL.size() ? AL.size() : end;
        SelectResult.toFile(  VeDate.getStringDate() +"  start="+start+" end="+end);
        futures.add(ex.submit(new Task(AL,issuesAll,start,end,a_numlimit)));
    }
    SelectResult.toFile(  VeDate.getStringDate() +" begin try and get Future" );
    try {
        //处理
        List<SsqIssue>  result = new ArrayList<>();
        for(Future<List<SsqIssue>> future : futures){
            //合并操作
            result.addAll(future.get());
        }
        SelectResult.toFile(  VeDate.getStringDate() +"   try end " );
        return result;
    } catch (Exception e) {
        e.printStackTrace();
        return null;
    }
    
}
 
private static class Task implements Callable<List<SsqIssue>>{	         
        private List<SsqIssue> AL;
        private List<SsqIssue> issuesAll;
        private int start;
        private int end;
        private int a_numlimit;
         
        public Task(List<SsqIssue> AL,List<SsqIssue> issuesAll,int start,int end,int a_numlimit){
        	
            this.AL = AL.subList(start, end);
            this.issuesAll=issuesAll;
            this.start = start;
            this.end = end;
            this.a_numlimit=a_numlimit;
        }
 
        public List<SsqIssue> call() throws Exception {
        	SelectResult.toFile(  VeDate.getStringDate() +"Task("+ AL.size()+" ,"+ issuesAll.size()+","+ start+","+ end+","+ a_numlimit+")" );
            //List<SsqIssue> retList = new ArrayList<SsqIssue>();
    		for(int i=0;i<AL.size();i++){
    			log("AL.size()="+AL.size());
    			for(int j=0;j<issuesAll.size();j++){				
    			
    				//log("Thread.currentThread().getName()="+Thread.currentThread().getName() +"|| AL.size()="+AL.size()); 
    				 SsqIssue SI=new  SsqIssue();
    				  SI=AL.get(i);
    				if(mycontains(SI.getRedBalls(),issuesAll.get(j).getRedBalls(),a_numlimit)){					
    					AL.remove(SI);
    				}				
    			  }				 
    		 }
            //返回处理结果
            return AL;
        }
    }

控制台日志如下：
MySsqStaticstic: AL.size()=275450
MySsqStaticstic: AL.size()=275449
MySsqStaticstic: AL.size()=275447
MySsqStaticstic: AL.size()=275447
MySsqStaticstic: AL.size()=275446
MySsqStaticstic: AL.size()=275445
MySsqStaticstic: AL.size()=275443
MySsqStaticstic: AL.size()=275441
MySsqStaticstic: AL.size()=275437
MySsqStaticstic: AL.size()=275434
MySsqStaticstic: AL.size()=275433
MySsqStaticstic: AL.size()=275431
MySsqStaticstic: AL.size()=275429
MySsqStaticstic: AL.size()=275427
MySsqStaticstic: AL.size()=275425
MySsqStaticstic: AL.size()=275424
...
...
怎么感觉还是单线程呀？？？从日志上看不出来是多线程并发在跑，为什么？ 分可以再加

[干饭人之路]

回楼上，对，我是想从AL中去掉与issueAll的重复数据。
目前写了一个优化方法如下：

//多线程处理AL
public static List<SsqIssue> dealListWithMutiThread(List<SsqIssue> AL,List<SsqIssue> issuesAll,int a_numlimit){
    //List<SsqIssue> list = new ArrayList<SsqIssue>(10000);
    
    int index = 0;
    int dealSize = 200000;
    int b=AL.size()/dealSize+1;
    ExecutorService ex = Executors.newFixedThreadPool(b);
    
   
    List<Future<List<SsqIssue>>> futures = new ArrayList<>();
    //log(" AL.size()/dealSize="+ AL.size()/dealSize);
    //sleep(10);
            //分配
    for(int i=0;i<= b;i++,index+=dealSize){
        int start = index;
        if(start>=AL.size()) break;
        int end = start + dealSize;
        end = end>AL.size() ? AL.size() : end;
        futures.add(ex.submit(new Task(AL,issuesAll,start,end,a_numlimit)));
    }
    try {
        //处理
        List<SsqIssue>  result = new ArrayList<>();
        for(Future<List<SsqIssue>> future : futures){
            //合并操作
            result.addAll(future.get());
        }
        return result;
    } catch (Exception e) {
        e.printStackTrace();
        return null;
    }
    
}
 
private static class Task implements Callable<List<SsqIssue>>{	         
        private List<SsqIssue> AL;
        private List<SsqIssue> issuesAll;
        private int start;
        private int end;
        private int a_numlimit;
         
        public Task(List<SsqIssue> AL,List<SsqIssue> issuesAll,int start,int end,int a_numlimit){
            this.AL = AL;
            this.issuesAll=issuesAll;
            this.start = start;
            this.end = end;
            this.a_numlimit=a_numlimit;
        }
 
        public List<SsqIssue> call() throws Exception {
            
    		for(int i=0;i<AL.size();i++){
                       //屏幕输出当前的AL.size()
    			log("AL.size()="+AL.size());
    			for(int j=0;j<issuesAll.size();j++){				
    			
    				//log("Thread.currentThread().getName()="+Thread.currentThread().getName() +"|| AL.size()="+AL.size()); 
    				
    				if(mycontains(AL.get(i).getRedBalls(),issuesAll.get(j).getRedBalls(),a_numlimit)){					
    					AL.remove(AL.get(i));
    				}				
    			  }				 
    		 }
            //返回处理结果
            return AL;
        }
    }

但在调用时遇到了新的问题，停在:
MySsqStaticstic: AL.size()=1306971
MySsqStaticstic: AL.size()=1306971
MySsqStaticstic: AL.size()=1306987
MySsqStaticstic: AL.size()=1306971
MySsqStaticstic: AL.size()=1306971
MySsqStaticstic: AL.size()=1306937
MySsqStaticstic: AL.size()=1306937
MySsqStaticstic: AL.size()=1306937
MySsqStaticstic: AL.size()=1306937
MySsqStaticstic: AL.size()=1306937
MySsqStaticstic: AL.size()=1306937

这里，就停在这里没有进一步运行了，为什么？何解？？？

[qybao]

你这是要达到一个什么样的效果？是AL去掉issueAll的重复数据吗？ 多线程涉及到同步（list的位置顺序不确定，数据还不好分段处理），估计改善不大。 可以重写MyIssue的equals和hashCode方法，利用集合提供的交并差（如removeAll）试试。

[qybao]

你只有275450条数据，你按500000条数据来开一个线程，AL.size/dealsize+1可不就是一个线程吗？

[干饭人之路]

楼上，//AL.size()=35678434

[effsim]

试试java8

[qq_39936465]

引用 楼主 臭鼬工厂 的回复:

		List<MyIssue> AL=new ArrayList<MyIssue>();
		List<MyIssue> issuesAll = new ArrayList();
		//此次略去N行代码给issuesAll和issuesAll赋值
		issuesAll = IssueDao.fromResultSet(resultAll);
		AL= xxxxx;
		
		 //issuesAll.size()=154345
		 //AL.size()=35678434 
		 //下面的两层嵌套循环总次数太多，请问各位有没有办法提高效率？？？？
		 //由于AL.size()远远大于issuesAll.size(),能否用多个进程并行？？？？代码怎么写？？
				for(int j=0;j<issuesAll.size();j++){										
						 for(int i=0;i<AL.size();i++){							
							if(mycontains(AL.get(i).getRedBalls(),issuesAll.get(j).getRedBalls(),a_numlimit)){								
									AL.remove(AL.get(i));								
								}				
						  }				 
					 }

为什么不贴 MyIssue类？？

[qq_39936465]

public static void main(String[] args) {
		// TODO Auto-generated method stub
		Map<Key, MyIssue> mapAL = new HashMap<>();//key 自己设定,key关键值得类型
		Map<Key, MyIssue> mapAll = new HashMap<>();
		List<MyIssue> AL=new ArrayList<MyIssue>();
		List<MyIssue> issuesAll = new ArrayList();
		AL.forEach(i->mapAL.put(key,MyIssue));
		issuesAll.forEach(i->mapAll.put(key,MyIssue));
		Iterator<Key> it1=mapAL.keySet().iterator();
		while(it1.hasNext()) {
			Key key= it1.next();
			if(mapAll.containsKey(key)) {
				mapAll.remove(key);
			}
		}
	}

[纵马饮白虹]

使用LinkedList遍历，遍历速度会更快
添加操作比删除操作更高效，所以使用第三个LinkedList，将不重复的数据添加到第三个list中