List txt, remove the two duplicates

Question

Hello.

Sorry if I'm not in the right place.
I have a list of lines in a txt file and I would like to remove all duplicate lines without leaving any of the lines that were duplicated, for example.

A
B
B
C
D
D

I would like it to give

A
C

I don't have Excel and I had already used my trial version with my Windows, so I would like to do this with free software like Notepad++. If anyone knows how to do it, please.

Thank you.

acefalo · Answer

Hello,  I can't test at the moment with Notepad ++ (I'm mainly on Linux), but I found this (Don't forget to work with a copy of the file):  You select the lines and in the Edit menu Edit->Line Operations->Sort Lines Lexicographically Ascending Edit->Line Operations->Remove Consecutive Duplicate Lines  I will give you an example using LibreOffice Calc, a free and open-source software. The process is longer, but I tested it and it works.  You start by copying your lines into column A.  In cell B1, you copy the following formula: =COUNTIF(A:A,A1). Then you drag the small black square at the bottom right of cell B1 all the way down. You will get the following result:  A 1 B 2 B 2 C 1 D 2 D 2  Then you click at the top of column A (to select it entirely) and then move to the right (keeping the left mouse button pressed) to also select column B entirely. Next, you go to the menu. Data/More Filters/Standard Filter  Field name: column B Condition: = Value: 1  You expand the Options menu. You check the box Copy results to: D1 You click on Validate.  You will have the following result: A 1 C 1  You can copy the unique column.

vinc · Answer

Hello acefalo. The manipulation with Notepad++ leaves me with one of the lines that was duplicated, so it doesn't work.

However, with LibreOffice Calc, it works very well. I'm amazed at the complexity of the manipulation; I wonder who comes up with stuff like that, lol.

However, when I have to drag the little black square at the bottom right of cell B1 to go all the way down, I wonder if there's a way to pull it down without dragging because if the list is long, it might get a bit complicated.

In any case, thank you very much, it helped me a lot.

acefalo · Answer

If you have access to a Linux distribution (this should also work on Windows, but perhaps in a different way).

file
A
B
B
C
D
D

In a terminal:

uniq -u file

A
C

To save:

uniq -u file > file2

A bit simpler than with LibreOffice! uniq is part of GNU coreutils, a set of tools that should be included by default in most distributions.

vinc · Answer

I'm sorry, but this time I didn't understand well.

Where should I click to the right of Fx? I can see Fx at the far right of the window
https://www.udrop.com/cache/plugins/filepreviewer/424319/28493bf178b27944f4c0add4e46465057494de78dce7c5cac1a5d15ad4b54aea/1100x800_cropped.jpg

I can't click to the right because there's nothing there, and when I click on it, it opens a function box where I have to choose between SUM, AVERAGE, MIN, MAX, or IF.

I also don't understand what you mean by writing B1:B5000 or another? In my case, the formula I need to put in B1 is =COUNTIF(A:A, A1), is that what I should put?

I'm really sorry, but I've never really used this kind of software before.

P.S.: Sorry, I just saw about Linux, I do have Linux on a VM, so I'll try it; it's true that it's much simpler this way. You really know a lot about this.
Thank you again.

acefalo · Answer

Hello,  I was doing two things at once (talking on the phone and writing). It should be to the left of Fx. At the top of the columns.

acefalo · Answer

Hello,

I don't know how many lines are in your file, but I did some testing just to see.

I started by creating a file with 100,000 pseudo-random numbers from 1 to 100,000 containing 10,000 unique numbers. I suspected that LibreOffice Calc wouldn't be able to give me results with the explained method (the list of unique numbers). I think Excel would do the same. Maybe by using another method. Calculating the frequency and then sorting instead of copying the cells. I managed to calculate the frequency for 100,000 with LibreOffice, but it took a few minutes. Maybe it would be better with a macro.

The same test with the command line (I add

time

in front of the command to calculate the execution time). It takes a fraction of a second for 100,000 numbers.

time sort file | uniq -u > file.out

real 0m0.183s
user 0m0.200s
sys 0m0.004s

I tested for 5,000 numbers with LibreOffice and it works quite well, although it's much slower.

Regarding my previous comment about line endings. I tested with different line endings (Windows, Unix, Classic Mac) for the

sort

and

uniq

commands. It seems to work for files with Windows line endings, but Mac line endings don't work right from the

sort

. It needs to be converted then.

To display line endings with Geany, you need to go to the View menu/Show Line Endings. To convert = Document menu/Set Line Endings. I imagine the same option is available with Notepad++. It's always better to work with a copy of the original file.

List txt, remove the two duplicates

6 réponses

How to get the character µ on the android keyboard.

Texture issues in enshrouded

Access issue to my "contact me" profile

Windows spotlight is not working.

Black screen digital camera

Connecting rj45 cable with fiber

Sony bravia tv red led blinks 4 times

Free: associating with the freebox server in progress

Inuyasha in french

Ccleaner starts at boot.