Limit lines in document to only so many characters, How?

None-yet

Member
Joined
Aug 10, 2020
Messages
78
Reaction score
32
Credits
906
I am not sure this even exists. I have a large wordlist file. Anyone who has worked with a wordlist will understand what I am going to ask here. This file is over 4 gb. I rarely open it. It has mixed in lines of very long text that appear to be random numbers and letters with like 100 characters. I would like to remove these using command line or something because to open the file takes a while and those lines are mixed here and there. I would take a good long effort to do by hand.

Does anyone know of a way I can delete these lines with a command line interface. I was thinking a command that would be able to limit each line to only a certain number of characters or something. If anyone knows how I can do this please also give me a breakdown of what you think will work. I am here to learn. Thanks
 


Do you have a few examples of words in that list that belong on the list and a few of those that you want removed.
 
Sorry for it taking so long. Client came in.

The ones that belong are just standard dictionary words. The line that don't belong. Below.


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

donesyq54c8s9w63w5s96w2z6w2q6s265q6ssddcd63c6dcd3c2d6ewd6df2d3363d36ew6fdf3ac33636d3333e3de3sa3JohnTheRipperf6f5vdfv1dgb4gfsd+c4+dsv4vsv4f4gb+fv+ g4bt+y4hr9t4hgf6 _1cxz3ca
 
Emacs or Vim can do it.
Make sure you have a backup copy of that file.
 
I greatly appreciate the answer. I went here and here. Didn't find an example of what I want to do. Plus with all of these you have to have the file open. This file being so large I am at the point of not wanting to open it because it takes so long. I was hoping for something that would take a command line and scan the document while it is closed and delete the unwanted lines. I may have to leave them in there but have never asked about doing this so I wanted to give this a shot.

Any suggestions?
 
That should give me something to go on. Did a quick look and it looks like it should work. Thank You Sir! If it doesn't or I need some help since I know very little about sed and awk I may post back. If anyone see's this that knows these post me a hello.

One question though. What would be a good start for me to study both sed and awk? I would love to read up on either or both to get a good understanding of them.

Thanks again gvisoc!
 
That should give me something to go on. Did a quick look and it looks like it should work. Thank You Sir! If it doesn't or I need some help since I know very little about sed and awk I may post back. If anyone see's this that knows these post me a hello.

One question though. What would be a good start for me to study both sed and awk? I would love to read up on either or both to get a good understanding of them.

Thanks again gvisoc!
Like the previous two posters said, vim, awk or sed should be able to do the job. My experience with them is limited so wouldn't know what to tell you. I would search do a google search with something around the lines of learn sed and awk.
 
I am actually trying to open the file now in vim to check something. However after 10 minutes it has not opened or it just may still be loading. It is a very large file. I am going to look through my library in a few to see if I already have something on sed and awk. I like the way awk works. Use it with Nmap.

Thanks
 
O-yea, just checked, I have UNIX Power Tools 2nd Addition sed & awk, The AWK Manual, Essential awk so these should get me going good.
 
What you are trying to do with the word list is possible with perl as well.
 
Last edited:
O-yea, just checked, I have UNIX Power Tools 2nd Addition sed & awk, The AWK Manual, Essential awk so these should get me going good.
I got some of those books too. I was going to recommend to have a look to humble bundle periodically, as sometimes they bundle (slightly old editions of) very good O’Reilly Unix books at ridiculously low prices.
 
you could write a PERL or Python script:
(pseudocode)
open file
foreach line in file
if linelength < 81
output to new file
end;

keith
 
I get digital books by the ftp full. I have an in with a few publishers so new or old they upload almost daily. I love reading and took a speed read class way back. I can read a 350 pg book in roughly two maybe three hours. It depends what else I have on my mind. I have four tb of books, novels, reference, white papers, manuals, text books-high schools, college, trade. Will not say I have everything but enough. So many I don't know what I have till I need something like the books I mentioned. I didn't know they were there. My mom said I was always weird with books. Reading newspapers when I was 5. My dad would quiz me on the news daily.

In case anyone needs or just enjoys reading and needs something I would be happy to provide if I am able.
 

Members online


Top