Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Best method for Pattern Matching on Binary String?

Programming languages, Coding, Executables, Package Creation, and Scripting.
Post Reply
Message
Author
starrysky1
Posts: 1
Joined: 2017-12-02 17:22

Best method for Pattern Matching on Binary String?

#1 Post by starrysky1 »

Hello and thanks in advanced.

I am in need of advice for the best language (was thinking SQL) or methodology for searching a long string of binary string data (couple of megabytes worth of binary digits at tops) which is being fed real time to some sort of text file or database for about one hundred thousand different (non permutations) patterns of binary digits.

I then need to display the most common pattern of all of these.

Its essentially a large block of binary digits that I need to search for a couple thousand different patterns in the fastest most effective way possible.

Any ideas as to what the best way to do this on real time data being fed to a text file would be? Does not really need to be exactly real time however I would rather it be as real time as possible. Id also rather it be a straight console affair so as to make it as latency free as possible however I can adapt as it doesn't really need to be terminal either.

Would a simple bash script be fast enough for 100,000 of such patterns? They are a maximum of 10 bits/character length patterns of 1's and 0's.

So for example:
10101110000110001011011111000101011010111101110100000011011101010101010111011100101010101011000101110011011111110001010010000110101011100000110011100101001010011001011010110101101010001010101010100010101010010100101001010101010111010101010010100101010011101010101010101001001010110101
and find whats the most common one out of these 100000 patterns there,

Also of note is that each new digit that comes in does so in about like 10 seconds time one after the other so its not like a huge new block of binary digits every time.

SQL really necessary/faster than just a straight bash script? All those thousands of patterns makes me think that just a simple bash script solution might be a little too slow and outdated. Ive heard about binary matching algorithms but I dont really know what those are and it seems a littlle out of my league at the moment, however I'm willing to get to the bottom if it if its truly the most effective way. What's the best way to do this ? Please advise; I'm essentially a programming noob and your advise is priceless to me right now.

So its just simply fast/effective pattern recognition on binary data. Any advice? I've got about 1 year of programming experience in Bash and am willing to work on whatever needs to be worked on in order to achieve this in the most effective way possible.

Doesn't look like it should be such an incredible feat to accomplish.

HAPPY HOLIDAYS!

User avatar
GarryRicketson
Posts: 5644
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: Best method for Pattern Matching on Binary String?

#2 Post by GarryRicketson »

Is this some kind of home work assignment ?

When I am looking for something , do this:
best language for searching a long string of binary string data

Another example:
Ive heard about binary matching algorithms but I dont really know what those are
what are binary matching algorithms ?
Doesn't look like it should be such an incredible feat to accomplish.
No, it isn't, so you shouldn't have any problem, or do you expect some one else to do it for you ? If it is not home work, why do you "need" to do this ? Work maybe ?...

http://www.tldp.org/LDP/abs/html/abs-guide.html
SQL really necessary/faster than just a straight bash script?
I don't think SQL will be of any use here.

Post Reply