Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Ok, I'll bite again:

  marek:~$ cat logs-popcount-org.txt | perf stat -d awk '!a[$0] { a[$0]=1; print }' > /dev/null 
  
   Performance counter stats for 'awk !a[$0] { a[$0]=1; print }':

           40,318.47 msec task-clock:u             
                   0      context-switches:u       
                   0      cpu-migrations:u         
           1,670,649      page-faults:u            
     112,979,634,215      cycles:u                 
      93,441,976,758      instructions:u           
      18,990,099,679      branches:u               
         208,386,137      branch-misses:u          
      26,093,832,363      L1-dcache-loads:u        
         708,880,979      L1-dcache-load-misses:u  
         464,332,790      LLC-loads:u              
         245,913,835      LLC-load-misses:u        

        40.337768657 seconds time elapsed
  
        36.851718000 seconds user
         3.468126000 seconds sys

Compare this to the optimized approach which has 57M LLC-load-misses, and 7M instructions.


I would welcome seeing a comparison in your environment to using the simple 1/2 GB array of bits, with no hashing or storage of IP addresses. (Extra points for hugetlb mapping.)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: