25 setembro, 2007
mailsys - a toolkit based on
When I first published gsinlistd (the co-operative anti-spam tool) I didn't realize it was so effective.
Started with v0.3-pre (indeed stable!), now I am continuing to work along with v0.5.
gMailSys started firstly with gsinlistd, now started to grow, and formed the mailsys toolkit.
The mailsys toolkit has a nice tool (gspamcompare) to measure spam. One of the interesting services of this tool is the so called "dig-service": it shows at once the SBL listings for IP(s).
This toolkit will be published soon; it uses the following libs:
Back to gspamcompare: take a look on the poa_check_ips.sh script output for my small server:
These are rats trying to drop spam, caught by greylistd; they are not polite, just close the TCP connection instead of issuing the QUIT command. By logging and processing these entries, alone, or again, in a co-operative way (like gsinlistd does by itself, and in addition!) it is possible to make short and real-time black-lists.
By the time I am writting this, I did run the same script in a big server online (from a friend) -- and caught two or three white-list entries. For those who know how greylist works, it is rather obvious greylist did its job, but now we have to enter with intrusive schemes like SpamAssassin. To avoid this extra work you can just monitor the Exim logs (say in each 15 mins) and black-list the unexpected rats.
This period is important: if it is too wide (beyond 60 mins, which is the usual greylist no-grace period) it is ineffective; if it is too narrow the process is too frantic searching for log dates. 15 min is fair and effective.
Removing these rats after one week of good behaviour seems rational. I.e., as long as the rats are no longer listed during the whole week, the black-list entries are removed; they have, however, to pass again to the regular grey-list stage. And the story comes back to the begining...
One great advantage: no real need of SpamAssassin for those kinds of unpolite rats.
Started with v0.3-pre (indeed stable!), now I am continuing to work along with v0.5.
gMailSys started firstly with gsinlistd, now started to grow, and formed the mailsys toolkit.
The mailsys toolkit has a nice tool (gspamcompare) to measure spam. One of the interesting services of this tool is the so called "dig-service": it shows at once the SBL listings for IP(s).
This toolkit will be published soon; it uses the following libs:
- libgmail
a library to dig mail boxes, and in general to deal with mail stuff; - liblowstream
the cyphered protocol used also by gsinlistd; - libgobj (refer to xpfweb_v2x)
- libgobjeio
(just an small lib for some dummy interfaces)
Back to gspamcompare: take a look on the poa_check_ips.sh script output for my small server:
These are rats trying to drop spam, caught by greylistd; they are not polite, just close the TCP connection instead of issuing the QUIT command. By logging and processing these entries, alone, or again, in a co-operative way (like gsinlistd does by itself, and in addition!) it is possible to make short and real-time black-lists.
By the time I am writting this, I did run the same script in a big server online (from a friend) -- and caught two or three white-list entries. For those who know how greylist works, it is rather obvious greylist did its job, but now we have to enter with intrusive schemes like SpamAssassin. To avoid this extra work you can just monitor the Exim logs (say in each 15 mins) and black-list the unexpected rats.
This period is important: if it is too wide (beyond 60 mins, which is the usual greylist no-grace period) it is ineffective; if it is too narrow the process is too frantic searching for log dates. 15 min is fair and effective.
Removing these rats after one week of good behaviour seems rational. I.e., as long as the rats are no longer listed during the whole week, the black-list entries are removed; they have, however, to pass again to the regular grey-list stage. And the story comes back to the begining...
One great advantage: no real need of SpamAssassin for those kinds of unpolite rats.
06 setembro, 2007
CNTP - the Civil Network Time Protocol
Some time ago I have started to study the clock drifts on VMware based guests (published on this article, in portuguese).
I have tried different approaches to make some (VMware) guests to synch their time correctly, but the clock behaviour was too erratic -- both hwclock command and system date were showing deviations, though in some guests the hwclock was a little more reliable.
I tried to use consecutive calls to ntpdate, the ntpd standard daemon, but nothing worked properly. ntpd protocol is too rigid and pedantic, after some time the ntpd just quitted.
CNTP first motivation is simplicity, second is to synch time. By this order.
CNTP stands for "Civil Network Time Protocol", a simple TCP service that has two basic entities: the server and the clients. It does not contain any stratums, authentication nor encryption, and by default is forkless -- this implies the server has no forks. Therefore clients may have to wait some time if the server is too busy attending requests, but the answer is relatively short and quick.
CNTP allows clock alignment with a second granularity: again, second goal is to synch time.
CNTP is a simple protocol that allows servers the optional knowledge for the local time, which depend on politics or logic behind DST (Daylight Saving Time).
Some specific protocol behaviour are derived from this protocol, the one covered in this article is VNTP. VNTP stands for "Virtual Network Time Protocol" -- suited especially for VMware guests, or machines with an unreliable clock.
The basic additional principle for VNTP is to have intelligent clients -- to minimize the charge of used VNTP servers. Nevertheless, on VMware guests it is expected to have lots of queries to the servers; the worse is the calculated clock drift the bigger is the number of queries.
VNTP clients have an additional important state called boundary. It varies typically between 1 and 9: one indicates a high drift (queries made on short periods), whilst 9 indicates a low drift (queries made on long periods, aligned to a minute boundary); the special value zero is used when the client does not use the minute boundary for alignment, but has no much of an interest.
The idea to avoid the minute boundary is to minimize the cases where the time adjustment (of HH:MM:SS + t seconds) makes SS seconds plus the calculated drift to change the system minute abruptly.
VNTP clients use always 'Ax' CNTP commands, usually x='b', so just 'Ab' CNTP commands; the length of commands sent is always 128 bytes.
Servers are identified as polite and non-polite; polite servers respond exactly as per CNTP protocol, whilst non-polite servers are those who refuse to connect, or provide long or short answers. Both are accepted by the client, as long as the response can be parsed.
VNTP clients never update the clock when the server is in the same second.
VNTP clients update the clock forward, or backwards; in the latter case they do it much slower -- motivation is to avoid an abrupt move of time back to the past. Usually VMware guests are delayed, their drift is rarely positive.
I have tried different approaches to make some (VMware) guests to synch their time correctly, but the clock behaviour was too erratic -- both hwclock command and system date were showing deviations, though in some guests the hwclock was a little more reliable.
I tried to use consecutive calls to ntpdate, the ntpd standard daemon, but nothing worked properly. ntpd protocol is too rigid and pedantic, after some time the ntpd just quitted.
CNTP first motivation is simplicity, second is to synch time. By this order.
CNTP stands for "Civil Network Time Protocol", a simple TCP service that has two basic entities: the server and the clients. It does not contain any stratums, authentication nor encryption, and by default is forkless -- this implies the server has no forks. Therefore clients may have to wait some time if the server is too busy attending requests, but the answer is relatively short and quick.
CNTP allows clock alignment with a second granularity: again, second goal is to synch time.
CNTP is a simple protocol that allows servers the optional knowledge for the local time, which depend on politics or logic behind DST (Daylight Saving Time).
Some specific protocol behaviour are derived from this protocol, the one covered in this article is VNTP. VNTP stands for "Virtual Network Time Protocol" -- suited especially for VMware guests, or machines with an unreliable clock.
The basic additional principle for VNTP is to have intelligent clients -- to minimize the charge of used VNTP servers. Nevertheless, on VMware guests it is expected to have lots of queries to the servers; the worse is the calculated clock drift the bigger is the number of queries.
VNTP clients have an additional important state called boundary. It varies typically between 1 and 9: one indicates a high drift (queries made on short periods), whilst 9 indicates a low drift (queries made on long periods, aligned to a minute boundary); the special value zero is used when the client does not use the minute boundary for alignment, but has no much of an interest.
The idea to avoid the minute boundary is to minimize the cases where the time adjustment (of HH:MM:SS + t seconds) makes SS seconds plus the calculated drift to change the system minute abruptly.
VNTP clients use always 'Ax' CNTP commands, usually x='b', so just 'Ab' CNTP commands; the length of commands sent is always 128 bytes.
Servers are identified as polite and non-polite; polite servers respond exactly as per CNTP protocol, whilst non-polite servers are those who refuse to connect, or provide long or short answers. Both are accepted by the client, as long as the response can be parsed.
VNTP clients never update the clock when the server is in the same second.
VNTP clients update the clock forward, or backwards; in the latter case they do it much slower -- motivation is to avoid an abrupt move of time back to the past. Usually VMware guests are delayed, their drift is rarely positive.
