initial

17 files changed, 2070 insertions, 0 deletions

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..7a0af4d
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+config.py
+*.pyc
+.idea/
diff --git a/LICENCE b/LICENCE
new file mode 100644
index 0000000..de9fd5a
--- /dev/null
+++ b/LICENCE
@@ -0,0 +1,502 @@
+		  GNU LESSER GENERAL PUBLIC LICENSE
+		       Version 2.1, February 1999
+
+ Copyright (C) 1991, 1999 Free Software Foundation, Inc.
+     51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+[This is the first released version of the Lesser GPL.  It also counts
+ as the successor of the GNU Library Public License, version 2, hence
+ the version number 2.1.]
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+Licenses are intended to guarantee your freedom to share and change
+free software--to make sure the software is free for all its users.
+
+  This license, the Lesser General Public License, applies to some
+specially designated software packages--typically libraries--of the
+Free Software Foundation and other authors who decide to use it.  You
+can use it too, but we suggest you first think carefully about whether
+this license or the ordinary General Public License is the better
+strategy to use in any particular case, based on the explanations below.
+
+  When we speak of free software, we are referring to freedom of use,
+not price.  Our General Public Licenses are designed to make sure that
+you have the freedom to distribute copies of free software (and charge
+for this service if you wish); that you receive source code or can get
+it if you want it; that you can change the software and use pieces of
+it in new free programs; and that you are informed that you can do
+these things.
+
+  To protect your rights, we need to make restrictions that forbid
+distributors to deny you these rights or to ask you to surrender these
+rights.  These restrictions translate to certain responsibilities for
+you if you distribute copies of the library or if you modify it.
+
+  For example, if you distribute copies of the library, whether gratis
+or for a fee, you must give the recipients all the rights that we gave
+you.  You must make sure that they, too, receive or can get the source
+code.  If you link other code with the library, you must provide
+complete object files to the recipients, so that they can relink them
+with the library after making changes to the library and recompiling
+it.  And you must show them these terms so they know their rights.
+
+  We protect your rights with a two-step method: (1) we copyright the
+library, and (2) we offer you this license, which gives you legal
+permission to copy, distribute and/or modify the library.
+
+  To protect each distributor, we want to make it very clear that
+there is no warranty for the free library.  Also, if the library is
+modified by someone else and passed on, the recipients should know
+that what they have is not the original version, so that the original
+author's reputation will not be affected by problems that might be
+introduced by others.
+^L
+  Finally, software patents pose a constant threat to the existence of
+any free program.  We wish to make sure that a company cannot
+effectively restrict the users of a free program by obtaining a
+restrictive license from a patent holder.  Therefore, we insist that
+any patent license obtained for a version of the library must be
+consistent with the full freedom of use specified in this license.
+
+  Most GNU software, including some libraries, is covered by the
+ordinary GNU General Public License.  This license, the GNU Lesser
+General Public License, applies to certain designated libraries, and
+is quite different from the ordinary General Public License.  We use
+this license for certain libraries in order to permit linking those
+libraries into non-free programs.
+
+  When a program is linked with a library, whether statically or using
+a shared library, the combination of the two is legally speaking a
+combined work, a derivative of the original library.  The ordinary
+General Public License therefore permits such linking only if the
+entire combination fits its criteria of freedom.  The Lesser General
+Public License permits more lax criteria for linking other code with
+the library.
+
+  We call this license the "Lesser" General Public License because it
+does Less to protect the user's freedom than the ordinary General
+Public License.  It also provides other free software developers Less
+of an advantage over competing non-free programs.  These disadvantages
+are the reason we use the ordinary General Public License for many
+libraries.  However, the Lesser license provides advantages in certain
+special circumstances.
+
+  For example, on rare occasions, there may be a special need to
+encourage the widest possible use of a certain library, so that it becomes
+a de-facto standard.  To achieve this, non-free programs must be
+allowed to use the library.  A more frequent case is that a free
+library does the same job as widely used non-free libraries.  In this
+case, there is little to gain by limiting the free library to free
+software only, so we use the Lesser General Public License.
+
+  In other cases, permission to use a particular library in non-free
+programs enables a greater number of people to use a large body of
+free software.  For example, permission to use the GNU C Library in
+non-free programs enables many more people to use the whole GNU
+operating system, as well as its variant, the GNU/Linux operating
+system.
+
+  Although the Lesser General Public License is Less protective of the
+users' freedom, it does ensure that the user of a program that is
+linked with the Library has the freedom and the wherewithal to run
+that program using a modified version of the Library.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.  Pay close attention to the difference between a
+"work based on the library" and a "work that uses the library".  The
+former contains code derived from the library, whereas the latter must
+be combined with the library in order to run.
+^L
+		  GNU LESSER GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License Agreement applies to any software library or other
+program which contains a notice placed by the copyright holder or
+other authorized party saying it may be distributed under the terms of
+this Lesser General Public License (also called "this License").
+Each licensee is addressed as "you".
+
+  A "library" means a collection of software functions and/or data
+prepared so as to be conveniently linked with application programs
+(which use some of those functions and data) to form executables.
+
+  The "Library", below, refers to any such software library or work
+which has been distributed under these terms.  A "work based on the
+Library" means either the Library or any derivative work under
+copyright law: that is to say, a work containing the Library or a
+portion of it, either verbatim or with modifications and/or translated
+straightforwardly into another language.  (Hereinafter, translation is
+included without limitation in the term "modification".)
+
+  "Source code" for a work means the preferred form of the work for
+making modifications to it.  For a library, complete source code means
+all the source code for all modules it contains, plus any associated
+interface definition files, plus the scripts used to control compilation
+and installation of the library.
+
+  Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running a program using the Library is not restricted, and output from
+such a program is covered only if its contents constitute a work based
+on the Library (independent of the use of the Library in a tool for
+writing it).  Whether that is true depends on what the Library does
+and what the program that uses the Library does.
+  
+  1. You may copy and distribute verbatim copies of the Library's
+complete source code as you receive it, in any medium, provided that
+you conspicuously and appropriately publish on each copy an
+appropriate copyright notice and disclaimer of warranty; keep intact
+all the notices that refer to this License and to the absence of any
+warranty; and distribute a copy of this License along with the
+Library.
+
+  You may charge a fee for the physical act of transferring a copy,
+and you may at your option offer warranty protection in exchange for a
+fee.
+^L
+  2. You may modify your copy or copies of the Library or any portion
+of it, thus forming a work based on the Library, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) The modified work must itself be a software library.
+
+    b) You must cause the files modified to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    c) You must cause the whole of the work to be licensed at no
+    charge to all third parties under the terms of this License.
+
+    d) If a facility in the modified Library refers to a function or a
+    table of data to be supplied by an application program that uses
+    the facility, other than as an argument passed when the facility
+    is invoked, then you must make a good faith effort to ensure that,
+    in the event an application does not supply such function or
+    table, the facility still operates, and performs whatever part of
+    its purpose remains meaningful.
+
+    (For example, a function in a library to compute square roots has
+    a purpose that is entirely well-defined independent of the
+    application.  Therefore, Subsection 2d requires that any
+    application-supplied function or table used by this function must
+    be optional: if the application does not supply it, the square
+    root function must still compute square roots.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Library,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Library, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote
+it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Library.
+
+In addition, mere aggregation of another work not based on the Library
+with the Library (or with a work based on the Library) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may opt to apply the terms of the ordinary GNU General Public
+License instead of this License to a given copy of the Library.  To do
+this, you must alter all the notices that refer to this License, so
+that they refer to the ordinary GNU General Public License, version 2,
+instead of to this License.  (If a newer version than version 2 of the
+ordinary GNU General Public License has appeared, then you can specify
+that version instead if you wish.)  Do not make any other change in
+these notices.
+^L
+  Once this change is made in a given copy, it is irreversible for
+that copy, so the ordinary GNU General Public License applies to all
+subsequent copies and derivative works made from that copy.
+
+  This option is useful when you wish to copy part of the code of
+the Library into a program that is not a library.
+
+  4. You may copy and distribute the Library (or a portion or
+derivative of it, under Section 2) in object code or executable form
+under the terms of Sections 1 and 2 above provided that you accompany
+it with the complete corresponding machine-readable source code, which
+must be distributed under the terms of Sections 1 and 2 above on a
+medium customarily used for software interchange.
+
+  If distribution of object code is made by offering access to copy
+from a designated place, then offering equivalent access to copy the
+source code from the same place satisfies the requirement to
+distribute the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  5. A program that contains no derivative of any portion of the
+Library, but is designed to work with the Library by being compiled or
+linked with it, is called a "work that uses the Library".  Such a
+work, in isolation, is not a derivative work of the Library, and
+therefore falls outside the scope of this License.
+
+  However, linking a "work that uses the Library" with the Library
+creates an executable that is a derivative of the Library (because it
+contains portions of the Library), rather than a "work that uses the
+library".  The executable is therefore covered by this License.
+Section 6 states terms for distribution of such executables.
+
+  When a "work that uses the Library" uses material from a header file
+that is part of the Library, the object code for the work may be a
+derivative work of the Library even though the source code is not.
+Whether this is true is especially significant if the work can be
+linked without the Library, or if the work is itself a library.  The
+threshold for this to be true is not precisely defined by law.
+
+  If such an object file uses only numerical parameters, data
+structure layouts and accessors, and small macros and small inline
+functions (ten lines or less in length), then the use of the object
+file is unrestricted, regardless of whether it is legally a derivative
+work.  (Executables containing this object code plus portions of the
+Library will still fall under Section 6.)
+
+  Otherwise, if the work is a derivative of the Library, you may
+distribute the object code for the work under the terms of Section 6.
+Any executables containing that work also fall under Section 6,
+whether or not they are linked directly with the Library itself.
+^L
+  6. As an exception to the Sections above, you may also combine or
+link a "work that uses the Library" with the Library to produce a
+work containing portions of the Library, and distribute that work
+under terms of your choice, provided that the terms permit
+modification of the work for the customer's own use and reverse
+engineering for debugging such modifications.
+
+  You must give prominent notice with each copy of the work that the
+Library is used in it and that the Library and its use are covered by
+this License.  You must supply a copy of this License.  If the work
+during execution displays copyright notices, you must include the
+copyright notice for the Library among them, as well as a reference
+directing the user to the copy of this License.  Also, you must do one
+of these things:
+
+    a) Accompany the work with the complete corresponding
+    machine-readable source code for the Library including whatever
+    changes were used in the work (which must be distributed under
+    Sections 1 and 2 above); and, if the work is an executable linked
+    with the Library, with the complete machine-readable "work that
+    uses the Library", as object code and/or source code, so that the
+    user can modify the Library and then relink to produce a modified
+    executable containing the modified Library.  (It is understood
+    that the user who changes the contents of definitions files in the
+    Library will not necessarily be able to recompile the application
+    to use the modified definitions.)
+
+    b) Use a suitable shared library mechanism for linking with the
+    Library.  A suitable mechanism is one that (1) uses at run time a
+    copy of the library already present on the user's computer system,
+    rather than copying library functions into the executable, and (2)
+    will operate properly with a modified version of the library, if
+    the user installs one, as long as the modified version is
+    interface-compatible with the version that the work was made with.
+
+    c) Accompany the work with a written offer, valid for at
+    least three years, to give the same user the materials
+    specified in Subsection 6a, above, for a charge no more
+    than the cost of performing this distribution.
+
+    d) If distribution of the work is made by offering access to copy
+    from a designated place, offer equivalent access to copy the above
+    specified materials from the same place.
+
+    e) Verify that the user has already received a copy of these
+    materials or that you have already sent this user a copy.
+
+  For an executable, the required form of the "work that uses the
+Library" must include any data and utility programs needed for
+reproducing the executable from it.  However, as a special exception,
+the materials to be distributed need not include anything that is
+normally distributed (in either source or binary form) with the major
+components (compiler, kernel, and so on) of the operating system on
+which the executable runs, unless that component itself accompanies
+the executable.
+
+  It may happen that this requirement contradicts the license
+restrictions of other proprietary libraries that do not normally
+accompany the operating system.  Such a contradiction means you cannot
+use both them and the Library together in an executable that you
+distribute.
+^L
+  7. You may place library facilities that are a work based on the
+Library side-by-side in a single library together with other library
+facilities not covered by this License, and distribute such a combined
+library, provided that the separate distribution of the work based on
+the Library and of the other library facilities is otherwise
+permitted, and provided that you do these two things:
+
+    a) Accompany the combined library with a copy of the same work
+    based on the Library, uncombined with any other library
+    facilities.  This must be distributed under the terms of the
+    Sections above.
+
+    b) Give prominent notice with the combined library of the fact
+    that part of it is a work based on the Library, and explaining
+    where to find the accompanying uncombined form of the same work.
+
+  8. You may not copy, modify, sublicense, link with, or distribute
+the Library except as expressly provided under this License.  Any
+attempt otherwise to copy, modify, sublicense, link with, or
+distribute the Library is void, and will automatically terminate your
+rights under this License.  However, parties who have received copies,
+or rights, from you under this License will not have their licenses
+terminated so long as such parties remain in full compliance.
+
+  9. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Library or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Library (or any work based on the
+Library), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Library or works based on it.
+
+  10. Each time you redistribute the Library (or any work based on the
+Library), the recipient automatically receives a license from the
+original licensor to copy, distribute, link with or modify the Library
+subject to these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties with
+this License.
+^L
+  11. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Library at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Library by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Library.
+
+If any portion of this section is held invalid or unenforceable under any
+particular circumstance, the balance of the section is intended to apply,
+and the section as a whole is intended to apply in other circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  12. If the distribution and/or use of the Library is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Library under this License may add
+an explicit geographical distribution limitation excluding those countries,
+so that distribution is permitted only in or among countries not thus
+excluded.  In such case, this License incorporates the limitation as if
+written in the body of this License.
+
+  13. The Free Software Foundation may publish revised and/or new
+versions of the Lesser General Public License from time to time.
+Such new versions will be similar in spirit to the present version,
+but may differ in detail to address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Library
+specifies a version number of this License which applies to it and
+"any later version", you have the option of following the terms and
+conditions either of that version or of any later version published by
+the Free Software Foundation.  If the Library does not specify a
+license version number, you may choose any version ever published by
+the Free Software Foundation.
+^L
+  14. If you wish to incorporate parts of the Library into other free
+programs whose distribution conditions are incompatible with these,
+write to the author to ask for permission.  For software which is
+copyrighted by the Free Software Foundation, write to the Free
+Software Foundation; we sometimes make exceptions for this.  Our
+decision will be guided by the two goals of preserving the free status
+of all derivatives of our free software and of promoting the sharing
+and reuse of software generally.
+
+			    NO WARRANTY
+
+  15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
+WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
+EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
+OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
+KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
+LIBRARY IS WITH YOU.  SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
+THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
+WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
+AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
+FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
+CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
+LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
+RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
+FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
+SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+^L
+           How to Apply These Terms to Your New Libraries
+
+  If you develop a new library, and you want it to be of the greatest
+possible use to the public, we recommend making it free software that
+everyone can redistribute and change.  You can do so by permitting
+redistribution under these terms (or, alternatively, under the terms of the
+ordinary General Public License).
+
+  To apply these terms, attach the following notices to the library.  It is
+safest to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least the
+"copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the library's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+
+Also add information on how to contact you by electronic and paper mail.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the library, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the
+  library `Frob' (a library for tweaking knobs) written by James Random Hacker.
+
+  <signature of Ty Coon>, 1 April 1990
+  Ty Coon, President of Vice
+
+That's all there is to it!
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..ddf6ac6
--- /dev/null
+++ b/README.md
@@ -0,0 +1,5 @@
+# Csound mailing list archives page
+
+The source of ml.csound.1bpm.net.
+
+Uses a NNTP server to store messages, a PostgreSQL server for message overviews and calculating thread membership etc, and a CGI script for presenting/rendering the pages.
diff --git a/base.css b/base.css
new file mode 100644
index 0000000..d044d3c
--- /dev/null
+++ b/base.css
@@ -0,0 +1,17 @@
+body {
+    font-family: Arial, Helvetica, sans-serif;
+}
+
+.tblodd {
+    background-color: #EEFFEE;
+}
+
+.tbleven {
+    background-color: #EEEEFF;
+}
+
+#tblmenu {
+    background-color: #EEEEFF;
+    font-weight: bold;
+    width: 100%;
+}
\ No newline at end of file
diff --git a/config.dist.py b/config.dist.py
new file mode 100644
index 0000000..c0fe702
--- /dev/null
+++ b/config.dist.py
@@ -0,0 +1,25 @@
+news_host = "in.bpm"
+
+lists = {
+    "mailinglist.csound": {
+        "mail_host": "",
+        "mail_user": "",
+        "mail_password": ""
+    },
+    "mailinglist.csound-dev": {
+        "mail_host": "",
+        "mail_user": "",
+        "mail_password": ""
+    },
+    "mailinglist.csound-tekno": None
+}
+
+db_host = ""
+db_name = ""
+db_user = ""
+db_password = ""
+
+rnews = "/usr/lib/news/bin/rnews -v"
+
+threads_per_page = 50
+
diff --git a/database_schema.sql b/database_schema.sql
new file mode 100644
index 0000000..5087b7c
--- /dev/null
+++ b/database_schema.sql
@@ -0,0 +1,323 @@
+--
+-- PostgreSQL database dump
+--
+
+-- Dumped from database version 14devel
+-- Dumped by pg_dump version 14devel
+
+SET statement_timeout = 0;
+SET lock_timeout = 0;
+SET idle_in_transaction_session_timeout = 0;
+SET client_encoding = 'UTF8';
+SET standard_conforming_strings = on;
+SELECT pg_catalog.set_config('search_path', '', false);
+SET check_function_bodies = false;
+SET xmloption = content;
+SET client_min_messages = warning;
+SET row_security = off;
+
+--
+-- Name: f_getthread(integer); Type: FUNCTION; Schema: public; Owner: kane
+--
+
+CREATE FUNCTION public.f_getthread(v_headid integer) RETURNS TABLE(id integer, messageid text, subject text, sender text, created timestamp with time zone, parentmessageid text)
+    LANGUAGE plpgsql
+    AS $$ begin
+return query
+with recursive src as (
+    select mp.* from message mp where mp.parentmessageid is null 
+and mp.id = f_getthreadhead(v_headid)
+    union all
+    select mc.* from message mc join src s on mc.parentmessageid = s.messageid
+), src2 as (select * from src)
+select * from src2 order by created asc;
+end; $$;
+
+
+ALTER FUNCTION public.f_getthread(v_headid integer) OWNER TO kane;
+
+--
+-- Name: f_getthread(text); Type: FUNCTION; Schema: public; Owner: kane
+--
+
+CREATE FUNCTION public.f_getthread(v_headid text) RETURNS TABLE(id integer, messageid text, subject text, sender text, created timestamp with time zone, parentmessageid text)
+    LANGUAGE plpgsql
+    AS $$ begin
+return query
+with recursive src as (
+    select mp.* from message mp where mp.parentmessageid is null 
+and mp.messageid = v_headid
+    union all
+    select mc.* from message mc join src s on mc.parentmessageid = s.messageid
+), src2 as (select * from src)
+select * from src2 order by created asc;
+end; $$;
+
+
+ALTER FUNCTION public.f_getthread(v_headid text) OWNER TO kane;
+
+--
+-- Name: f_getthread(text, integer); Type: FUNCTION; Schema: public; Owner: kane
+--
+
+CREATE FUNCTION public.f_getthread(v_mailinglist text, v_headid integer) RETURNS TABLE(id integer, messageid text, subject text, sender text, created timestamp with time zone, parentmessageid text, mailinglist_id integer)
+    LANGUAGE plpgsql
+    AS $$ 
+declare
+	v_mailinglist_id int;
+begin
+
+select ml.id into v_mailinglist_id from mailinglist ml where name = v_mailinglist;
+
+return query
+with recursive src as (
+    select 
+		mp.id, mp.messageid, mp.subject, mp.sender, mp.created, mp.parentmessageid, mp.mailinglist_id
+	from message mp 
+	where 1=1 --mp.parentmessageid is null 
+		and mp.id = f_getthreadhead(v_headid)
+		and mp.mailinglist_id = v_mailinglist_id
+
+	union all select 
+		mc.id, mc.messageid, mc.subject, mc.sender, mc.created, mc.parentmessageid, s.mailinglist_id
+	from message mc 
+	join src s 
+		on mc.parentmessageid = s.messageid
+), src2 as (select * from src)
+select * from src2 order by created asc;
+end; $$;
+
+
+ALTER FUNCTION public.f_getthread(v_mailinglist text, v_headid integer) OWNER TO kane;
+
+--
+-- Name: f_getthreadhead(integer); Type: FUNCTION; Schema: public; Owner: kane
+--
+
+CREATE FUNCTION public.f_getthreadhead(v_id integer) RETURNS integer
+    LANGUAGE plpgsql
+    AS $$ declare v_res int; begin
+
+with recursive src as (
+	select * from message where id = v_id
+	union all
+	select 
+		m.* 
+	from message m 
+	join src s on s.parentmessageid = m.messageid
+)
+select id into v_res from src order by created asc limit 1; --where parentmessageid is null;
+return v_res;
+end; $$;
+
+
+ALTER FUNCTION public.f_getthreadhead(v_id integer) OWNER TO kane;
+
+--
+-- Name: f_getthreadids(text); Type: FUNCTION; Schema: public; Owner: kane
+--
+
+CREATE FUNCTION public.f_getthreadids(v_headid text) RETURNS TABLE(id integer)
+    LANGUAGE plpgsql
+    AS $$ begin
+return query
+with recursive src as (
+    select mp.* from message mp where mp.parentmessageid is null 
+and mp.messageid = v_headid
+    union all
+    select mc.* from message mc join src s on mc.parentmessageid = s.messageid
+), src2 as (select * from src)
+select src2.id from src2 order by created asc;
+end; $$;
+
+
+ALTER FUNCTION public.f_getthreadids(v_headid text) OWNER TO kane;
+
+SET default_tablespace = '';
+
+SET default_table_access_method = heap;
+
+--
+-- Name: mailinglist; Type: TABLE; Schema: public; Owner: kane
+--
+
+CREATE TABLE public.mailinglist (
+    id integer NOT NULL,
+    name text,
+    description text
+);
+
+
+ALTER TABLE public.mailinglist OWNER TO kane;
+
+--
+-- Name: mailinglist_id_seq; Type: SEQUENCE; Schema: public; Owner: kane
+--
+
+CREATE SEQUENCE public.mailinglist_id_seq
+    AS integer
+    START WITH 1
+    INCREMENT BY 1
+    NO MINVALUE
+    NO MAXVALUE
+    CACHE 1;
+
+
+ALTER TABLE public.mailinglist_id_seq OWNER TO kane;
+
+--
+-- Name: mailinglist_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: kane
+--
+
+ALTER SEQUENCE public.mailinglist_id_seq OWNED BY public.mailinglist.id;
+
+
+--
+-- Name: message; Type: TABLE; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE TABLE public.message (
+    id integer NOT NULL,
+    messageid text,
+    subject text,
+    sender text,
+    created timestamp with time zone,
+    parentmessageid text,
+    mailinglist_id integer
+);
+
+
+ALTER TABLE public.message OWNER TO csoundmailinglist;
+
+--
+-- Name: message_id_seq; Type: SEQUENCE; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE SEQUENCE public.message_id_seq
+    AS integer
+    START WITH 1
+    INCREMENT BY 1
+    NO MINVALUE
+    NO MAXVALUE
+    CACHE 1;
+
+
+ALTER TABLE public.message_id_seq OWNER TO csoundmailinglist;
+
+--
+-- Name: message_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: csoundmailinglist
+--
+
+ALTER SEQUENCE public.message_id_seq OWNED BY public.message.id;
+
+
+--
+-- Name: messagetext; Type: TABLE; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE TABLE public.messagetext (
+    id integer NOT NULL,
+    message_id integer,
+    line integer,
+    data tsvector
+);
+
+
+ALTER TABLE public.messagetext OWNER TO csoundmailinglist;
+
+--
+-- Name: messagetext_id_seq; Type: SEQUENCE; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE SEQUENCE public.messagetext_id_seq
+    AS integer
+    START WITH 1
+    INCREMENT BY 1
+    NO MINVALUE
+    NO MAXVALUE
+    CACHE 1;
+
+
+ALTER TABLE public.messagetext_id_seq OWNER TO csoundmailinglist;
+
+--
+-- Name: messagetext_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: csoundmailinglist
+--
+
+ALTER SEQUENCE public.messagetext_id_seq OWNED BY public.messagetext.id;
+
+
+--
+-- Name: mailinglist id; Type: DEFAULT; Schema: public; Owner: kane
+--
+
+ALTER TABLE ONLY public.mailinglist ALTER COLUMN id SET DEFAULT nextval('public.mailinglist_id_seq'::regclass);
+
+
+--
+-- Name: message id; Type: DEFAULT; Schema: public; Owner: csoundmailinglist
+--
+
+ALTER TABLE ONLY public.message ALTER COLUMN id SET DEFAULT nextval('public.message_id_seq'::regclass);
+
+
+--
+-- Name: messagetext id; Type: DEFAULT; Schema: public; Owner: csoundmailinglist
+--
+
+ALTER TABLE ONLY public.messagetext ALTER COLUMN id SET DEFAULT nextval('public.messagetext_id_seq'::regclass);
+
+
+--
+-- Name: ix_mailinglist; Type: INDEX; Schema: public; Owner: kane
+--
+
+CREATE INDEX ix_mailinglist ON public.mailinglist USING btree (id, name);
+
+
+--
+-- Name: ix_message_id; Type: INDEX; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE INDEX ix_message_id ON public.message USING btree (id);
+
+
+--
+-- Name: ix_message_messageid; Type: INDEX; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE INDEX ix_message_messageid ON public.message USING btree (messageid, parentmessageid, subject, sender, created, mailinglist_id);
+
+
+--
+-- Name: ix_message_parent; Type: INDEX; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE INDEX ix_message_parent ON public.message USING btree (parentmessageid);
+
+
+--
+-- Name: ix_messagetext_ix; Type: INDEX; Schema: public; Owner: csoundmailinglist
+--
+
+CREATE INDEX ix_messagetext_ix ON public.messagetext USING btree (message_id, data);
+
+
+--
+-- Name: TABLE mailinglist; Type: ACL; Schema: public; Owner: kane
+--
+
+GRANT SELECT ON TABLE public.mailinglist TO csoundmailinglist;
+
+
+--
+-- Name: SEQUENCE mailinglist_id_seq; Type: ACL; Schema: public; Owner: kane
+--
+
+GRANT USAGE ON SEQUENCE public.mailinglist_id_seq TO csoundmailinglist;
+
+
+--
+-- PostgreSQL database dump complete
+--
+
diff --git a/db.py b/db.py
new file mode 100644
index 0000000..19e194b
--- /dev/null
+++ b/db.py
@@ -0,0 +1,37 @@
+import config
+from peewee import *
+from playhouse.postgres_ext import *
+
+
+_db = PostgresqlExtDatabase(config.db_name, autorollback=True, **{
+    "host": config.db_host,
+    "password": config.db_password,
+    "user": config.db_user
+})
+
+
+class BaseModel(Model):
+    class Meta:
+        database = _db
+
+class MailingList(BaseModel):
+    name = TextField()
+    description = TextField()
+
+class Message(BaseModel):
+    messageid = TextField()
+    subject = TextField()
+    sender = TextField()
+    created = TextField()
+    parentmessageid = TextField()
+    mailinglist = ForeignKeyField(MailingList)
+
+
+class MessageText(BaseModel):
+    message = ForeignKeyField(Message)
+    line = IntegerField()
+    data = TSVectorField()
+
+
+def housekeeping():
+    _db.execute_sql("delete from messagetext where data = '';")
diff --git a/emailprocessor.py b/emailprocessor.py
new file mode 100644
index 0000000..4ff27e7
--- /dev/null
+++ b/emailprocessor.py
@@ -0,0 +1,33 @@
+import imaplib
+import email
+import email.header
+
+
+class EmailProcessor(object):
+
+    def __init__(self, process_func, host, username, password, mailbox="INBOX", search="ALL", delete=True):
+        self.processed_number = 0
+        server = imaplib.IMAP4(host)
+        status, detail = server.login_cram_md5(username, password)
+        if status != "OK":
+            raise Exception("Login failed")
+        status, num = server.select(mailbox)
+        if status != "OK":
+            raise Exception("Selecting mailbox {} failed".format(mailbox))
+        status, data = server.search(None, search)
+        if status != "OK":
+            raise Exception("Obtaining message list failed")
+        for num in data[0].split():
+            status, data = server.fetch(num, "(RFC822)")
+            if status != "OK":
+                raise Exception("Obtaining message {} failed".format(num))
+            msg = email.message_from_string(data[0][1])
+            result = process_func(msg)
+            if delete and result is not False:
+                server.store(num, "+FLAGS", "\\Deleted")
+            self.processed_number += 1
+        server.expunge()
+        server.close()
+        server.logout()
+
+
diff --git a/generate_static.py b/generate_static.py
new file mode 100644
index 0000000..0c5d838
--- /dev/null
+++ b/generate_static.py
@@ -0,0 +1,402 @@
+import db
+import config
+import email
+import email.utils
+import nntplib
+import os
+import sys
+import cgi
+import lxml.html
+import math
+
+_nntp = None
+goutput_path = None
+
+
+def get_nntp():
+    global _nntp
+    if not _nntp:
+        _nntp = nntplib.NNTP(config.news_host, readermode=True)
+        _nntp.group(config.newsgroup)
+    return _nntp
+
+def close_nntp():
+    if not _nntp:
+        return
+    _nntp.quit()
+
+
+class AttachedFile(object):
+    def __init__(self, name, payload):
+        self.name = name
+        self.payload = payload
+
+class ParsedMessage(object):
+    def __init__(self, string):
+        self.html = unicode()
+        self.text = unicode()
+        self.attachments = list()
+        msg = email.message_from_string(string)
+        self.parse(msg)
+
+    def clean_html(self):
+        root = lxml.html.fromstring(self.html)
+        body_found = False
+        output = unicode()
+        head = root.findall(".//head")
+        if head:
+            head = head[0]
+            output += (head.text or u"") + u"".join([lxml.html.tostring(child) for child in head.iterchildren()])
+
+        body = root.findall(".//body")
+        if body:
+            body = body[0]
+            body_found = True
+            output += (body.text or u"") + u"".join([lxml.html.tostring(child) for child in body.iterchildren()])
+
+        if body_found:
+            self.html = output
+
+    def parse(self, parts):
+        for part in parts.walk():
+            ct = part.get_content_type()
+            cd = part.get("Content-Disposition")
+            if cd:
+                filename = part.get_filename()
+                self.attachments.append(AttachedFile(filename, part.get_payload(decode=True)))
+            elif ct == "text/plain":
+                charset = part.get_content_charset("utf-8")
+                self.text += part.get_payload(decode=True).decode(charset)
+            elif ct == "text/html":
+                charset = part.get_content_charset("utf-8")
+                self.html += part.get_payload(decode=True).decode(charset)
+            elif ct.startswith("multipart"):
+                for p in part.get_payload():
+                    pass #self.parse(p)
+        try:
+            self.clean_html()
+        except:
+            pass
+
+        
+
+def name_from_email(address):
+    try:
+        parsed = email.utils.parseaddr(address)
+        decoded = email.Header.decode_header(parsed[0])[0]
+        return decoded[0].decode(decoded[1])
+    except:
+        return address
+
+
+class Message(object):  # raw workaround for non-peewee table function hackery
+    def __init__(self, db_msg=None, raw_msg=None):
+        self.db_msg = db_msg
+        self.raw_msg = raw_msg
+        self.parsed_article = None
+
+    @property
+    def id(self):
+        return self.raw_msg[0] if self.raw_msg else self.db_msg.id
+
+
+    @property
+    def sender(self):
+        return self.raw_msg[3] if self.raw_msg else self.db_msg.sender
+
+    @property
+    def sender_name(self):
+        return name_from_email(self.sender)
+
+    @property
+    def messageid(self):
+        return self.raw_msg[1] if self.raw_msg else self.db_msg.messageid
+
+    @property
+    def subject(self):
+        subject = self.raw_msg[2] if self.raw_msg else self.db_msg.subject
+        try:
+            return email.Header.decode_header(subject)[0][0]
+        except:
+            return subject
+
+    @property
+    def created(self):
+        return self.raw_msg[4] if self.raw_msg else self.db_msg.created
+
+    @property
+    def parentmessageid(self):
+        return self.raw_msg[5] if self.raw_msg else self.db_msg.parentmessageid
+
+    @property
+    def article(self):
+        return self.parsed_article
+
+    def parse_article(self):
+        result = get_nntp().article(self.messageid)
+        #msg = u"\n".join([x.decode("utf-8") for x in result[3]])
+        msg = "\n".join(result[3])
+        self.parsed_article = ParsedMessage(msg)
+
+
+    @property
+    def html(self):
+        content = u"<div id=\"message{}\"><table class=\"messagetable\"><tbody>".format(self.id) +\
+            u"<tr><td><b>Date</b></td><td>{}</td></tr>".format(str(self.created)[:16]) +\
+            u"<tr><td><b>From</b></td><td>{}</td></tr>".format(self.sender_name) +\
+            u"<tr><td><b>Subject</b></td><td>{}</td></tr>".format(self.subject)
+
+        if len(self.article.attachments) > 0:
+            content += u"<tr><td><b>Attachments</b></td><td>"
+            for index, attach in enumerate(self.article.attachments):
+                fname = u"{0}.{1}.{2}".format(self.id, index, attach.name)
+                output_path = os.path.join(goutput_path, u"attachment", fname)
+                content += u"<a href=\"/attachment/{}\">{}</a>&nbsp;&nbsp;".format(fname, attach.name)
+            content += u"</td></tr>"
+
+        content += u"<tr><td></td><td>"
+        if self.article.html:
+            content += self.article.html
+        elif self.article.text:
+            content += u"<pre>{}</pre>".format(self.article.text)
+
+        content += u"</td></tr></tbody></table></div><hr />"
+        return content
+
+    def get_dict(self, with_article):
+        data = {
+            "id": self.id,
+            "sender": self.sender,
+            "messageid": self.messageid,
+            "subject": self.subject,
+            "created": self.created,
+            "parentmessageid": self.parentmessageid
+        }
+        if with_article:
+            data["article"] = self.article
+        return data
+
+
+def get_thread_heads(page=1, page_size=50, year=None, month=None):
+    return db.Message.select().where(
+        (db.Message.parentmessageid.is_null())
+    ).order_by(db.Message.created.desc()).paginate(page, page_size)
+
+
+
+
+class Page(object):
+    def __init__(self):
+        self.content = unicode()
+        self.title = unicode()
+
+    def create(self, data=None):
+        pass
+
+    def render(self, data=None):
+        self.create(data)
+        with open(u"templates/base.html", "r") as f:
+            base = unicode(f.read())
+        page = base.replace(u"V_TITLE", self.title).replace(u"V_CONTENT", self.content)
+        print u"Content-Type: text/html; charset=utf-8\r\n\r\n"
+        sys.stdout.write(page.encode("utf-8"))
+        close_nntp()
+        #print page
+
+
+class Overview(Page):
+    def create(self, data=None):
+        page = data.get("page") or 1 if data else 1
+        self.title = u"Csound mailing list archive, page {}".format(page)
+
+        pagination = unicode()
+        if page != 1:
+            pagination += u"<a href=\"/page/{}\">Previous</a>&nbsp;".format(page-1)
+        for x in range(max(1, page-10), page+10):
+            if x == page:
+                pagination += u"{} ".format(x)
+            else:
+                pagination += u"<a href=\"/page/{0}\">{0}</a>&nbsp;".format(x)
+        pagination += u"<a href=\"/page/{}\">Next</a>&nbsp;".format(page+1)
+
+        threads = get_thread_heads(page, config.threads_per_page)
+        content = u"<h2>Csound mailing list archive</h2><h3>Page {}</h3>{}".format(page, pagination) +\
+            u"<table id=\"threads\"><thead><tr><th>Date</th><th>From</th><th>Subject</th></tr></thead><tbody>"
+        odd = True
+        for thread in threads:
+            tmessage = Message(db_msg=thread)
+            content += u"<tr class=\"{}\"><td>{}</td><td>{}</td><td><a href=\"/thread/{}\">{}</a></td></tr>".format(
+                "tbl{}".format("odd" if odd else "even"),
+                unicode(tmessage.created)[:16],
+                tmessage.sender_name,
+                thread.id,
+                tmessage.subject
+            )
+            odd = not odd
+
+
+        content += u"</tbody></table><div id=\"pagination\">{}</div>".format(pagination)
+        self.content = content
+
+
+class Thread(Page):
+    def create(self, data=None):
+        content = unicode()
+        # nasty hack, parameter passing is not working, ugh, but int at least
+        sql = "SELECT id, messageid, subject, sender, created, parentmessageid FROM f_getthread({});".format(int(data["threadid"]))
+        cursor = db._db.execute_sql(sql)
+        first = True
+        for item in cursor:
+            msg = Message(raw_msg=item)
+            msg.parse_article()
+            if first:
+                first = False
+                subject = msg.subject
+                content = u"<h2>{}</h2>".format(subject)
+                self.title = subject
+            content += msg.html
+
+        cursor.close()
+        self.content += content
+
+
+
+
+class SearchResult(Page):
+    def create(self, data=None):
+        form = cgi.FieldStorage()
+        term = form.getvalue("term")
+
+        if not term:
+            self.title = "Search the Csound mailing list archive"
+            with open("templates/search.html", "r") as f:
+                self.content = f.read()
+            return
+
+        title = u"Search results for '{}'".format(term)
+        self.title = title
+        content = u"<h2>{}</h2>".format(title) +\
+            u"<table id=\"threads\"><thead><tr><th>Date</th><th>From</th><th>Subject</th></tr></thead><tbody>"
+
+        if form.getvalue("searchbody"):
+            query = db.Message.select(db.Message).where(
+                (db.Message.subject.contains(term) if form.getvalue("searchsubject") else True)
+                | (db.Message.sender.contains(term) if form.getvalue("searchsender") else True)
+                | (db.MessageText.data.match(term) if form.getvalue("searchbody") else True)
+            ).join(db.MessageText, join_type=db.JOIN.LEFT_OUTER).distinct().order_by(db.Message.created.desc()).limit(200)
+        else:
+            query = db.Message.select().where(
+                (db.Message.subject.contains(term) if form.getvalue("searchsubject") else True)
+                | (db.Message.sender.contains(term) if form.getvalue("searchsender") else True)
+            ).distinct().order_by(db.Message.created.desc()).limit(200)
+
+        odd = True
+        for message in query:
+            tmessage = Message(db_msg=message)
+            content += u"<tr class=\"{0}\"><td>{1}</td><td>{2}</td><td><a href=\"/thread/{3}#message{3}\">{4}</a></td></tr>".format(
+                "tbl{}".format("odd" if odd else "even"),
+                unicode(tmessage.created)[:16],
+                tmessage.sender_name,
+                message.id,
+                tmessage.subject
+            )
+            odd = not odd
+
+        content += u"</tbody></table>"
+        self.content = content
+
+
+
+def handle():
+    path = os.environ.get("REQUEST_URI")
+    if not path or path == "/":
+        return Overview().render()
+
+    parts = path.split("/")[1:]
+    rtype = parts[0]
+
+    if rtype == "thread":
+        return Thread().render({"threadid": int(parts[1])})
+    elif rtype == "page":
+        return Overview().render({"page": int(parts[1])})
+    elif rtype == "search":
+        return SearchResult().render()
+    elif rtype == "about":
+        return About().render()
+    elif rtype == "attachment":
+        return Attachment().render({"ident": parts[1]})
+
+    return Overview().render()
+
+
+def generate_indexpage(page_base, output_path, page, max_page, data):
+    title = u"Csound mailing list archive, page {}".format(page)
+    output_file = os.path.join(output_path, u"page", u"{}.html".format(page))
+    pagination = unicode()
+
+    if page != 1:
+        pagination += u"<a href=\"/page/{}.html\">Previous</a>&nbsp;".format(page-1)
+    for x in range(max(1, page-10), min(page+10, max_page)):
+        if x == page:
+            pagination += u"{} ".format(x)
+        else:
+            pagination += u"<a href=\"/page/{0}.html\">{0}</a>&nbsp;".format(x)
+
+    if page != max_page:
+        pagination += u"<a href=\"/page/{}.html\">Next</a>&nbsp;".format(page+1)
+
+    content = u"<h2>Csound mailing list archive</h2><h3>Page {}</h3>{}".format(page, pagination) +\
+        u"<table id=\"threads\"><thead><tr><th>Date</th><th>From</th><th>Subject</th></tr></thead><tbody>"
+    odd = True
+    for thread in data:
+        content += u"<tr class=\"{}\"><td>{}</td><td>{}</td><td><a href=\"/thread/{}.html\">{}</a></td></tr>".format(
+            "tbl{}".format("odd" if odd else "even"),
+            unicode(thread.created)[:16],
+            name_from_email(thread.sender),
+            thread.id,
+            thread.subject
+        )
+        odd = not odd
+
+    content += u"</tbody></table><div id=\"pagination\">{}</div>".format(pagination)
+    html = page_base.replace(u"V_TITLE", title).replace(u"V_CONTENT", content)
+    with open(output_file, "w") as f:
+        f.write(html.encode("utf-8"))
+
+
+
+def generator(output_path):
+    global goutput_path
+    goutput_path = output_path
+    with open(u"templates/base.html", "r") as f:
+        page_base = unicode(f.read())
+    total_messages = db.Message.select().where(db.Message.parentmessageid.is_null()).count()
+    max_page = math.ceil(float(total_messages) / config.threads_per_page)
+    threads = db.Message.select().where(db.Message.parentmessageid.is_null())
+
+    thread_number = 1
+    index_page = 1
+    overview = list()
+    for thread in threads:
+        try:
+            overview.append(thread)
+            tdata = Thread()
+            tdata.create({"threadid": thread.id})
+            html = page_base.replace(u"V_TITLE", tdata.title).replace(u"V_CONTENT", tdata.content)
+
+            output_file = os.path.join(output_path, u"thread", u"{}.html".format(thread.id))
+            with open(output_file, "w") as f:
+                f.write(html.encode("utf-8"))
+
+            if thread_number == config.threads_per_page:
+                thread_number = 0
+            else:
+                index_page += 1
+                thread_number += 1
+        except Exception, ex:
+            print ex
+
+
+if __name__ == "__main__":
+    generator(unicode(sys.argv[1]))
+
diff --git a/handler.py b/handler.py
new file mode 100755
index 0000000..ad185c2
--- /dev/null
+++ b/handler.py
@@ -0,0 +1,390 @@
+#!/usr/bin/env python
+import db
+import config
+import email
+import email.utils
+import nntplib
+import os
+import sys
+import cgi
+import lxml.html
+
+_nntp = None
+
+
+
+def get_nntp(mailing_list):
+    global _nntp
+    if not _nntp:
+        if mailing_list not in config.lists:
+            raise Exception("Mailing list does not exist")
+        _nntp = nntplib.NNTP(config.news_host, readermode=True)
+        _nntp.group(mailing_list)
+    return _nntp
+
+def close_nntp():
+    if not _nntp:
+        return
+    _nntp.quit()
+
+
+class ParsedMessage(object):
+    def __init__(self, string, keep_attachment=None):
+        self.html = unicode()
+        self.text = unicode()
+        self.keep_attachment = keep_attachment
+        self.attachments = list()
+        self.attachment = None
+        msg = email.message_from_string(string)
+        self.parse(msg)
+
+    def clean_html(self):
+        root = lxml.html.fromstring(self.html)
+        body_found = False
+        output = unicode()
+        head = root.findall(".//head")
+        if head:
+            head = head[0]
+            output += (head.text or "") + "".join([lxml.html.tostring(child) for child in head.iterchildren()])
+
+        body = root.findall(".//body")
+        if body:
+            body = body[0]
+            body_found = True
+            output += (body.text or "") + "".join([lxml.html.tostring(child) for child in body.iterchildren()])
+
+        if body_found:
+            self.html = output
+
+    
+    def parse(self, parts):
+        anum = 0
+        for part in parts.walk():
+            ct = part.get_content_type()
+            cd = part.get("Content-Disposition")
+            if cd:
+                filename = part.get_filename()
+                self.attachments.append(filename)
+                if self.keep_attachment == anum:
+                    self.attachment = "Content-Type:{}\r\nContent-Disposition:{}\r\n\r\n".format(
+                        ct, cd
+                    )
+                    self.attachment += part.get_payload(decode=True)
+                anum += 1
+            elif ct == "text/plain":
+                charset = part.get_content_charset("utf-8")
+                self.text += part.get_payload(decode=True).decode(charset)
+            elif ct == "text/html":
+                charset = part.get_content_charset("utf-8")
+                self.html += part.get_payload(decode=True).decode(charset)
+            elif ct.startswith("multipart"):
+                for p in part.get_payload():
+                    pass #self.parse(p)
+        try:
+            self.clean_html()
+        except:
+            pass
+
+        
+
+
+
+class Message(object):  # raw workaround for non-peewee table function hackery
+    def __init__(self, mailing_list, db_msg=None, raw_msg=None):
+        self.mailing_list = mailing_list
+        self.db_msg = db_msg
+        self.raw_msg = raw_msg
+        self.parsed_article = None
+
+
+    @property
+    def id(self):
+        return self.raw_msg[0] if self.raw_msg else self.db_msg.id
+
+
+    @property
+    def sender(self):
+        return self.raw_msg[3] if self.raw_msg else self.db_msg.sender
+
+    @property
+    def sender_name(self):
+        try:
+            parsed = email.utils.parseaddr(self.sender)
+            decoded = email.Header.decode_header(parsed[0])[0]
+            return decoded[0].decode(decoded[1])
+        except:
+            return self.sender
+
+    @property
+    def messageid(self):
+        return self.raw_msg[1] if self.raw_msg else self.db_msg.messageid
+
+    @property
+    def subject(self):
+        subject = self.raw_msg[2] if self.raw_msg else self.db_msg.subject
+        try:
+            return email.Header.decode_header(subject)[0][0]
+        except:
+            return subject
+
+    @property
+    def created(self):
+        return self.raw_msg[4] if self.raw_msg else self.db_msg.created
+
+    @property
+    def parentmessageid(self):
+        return self.raw_msg[5] if self.raw_msg else self.db_msg.parentmessageid
+
+    @property
+    def article(self):
+        if not self.parsed_article:
+            result = get_nntp(self.mailing_list).article(self.messageid)
+            #msg = u"\n".join([x.decode("utf-8") for x in result[3]])
+            msg = "\n".join(result[3])
+            self.parsed_article = ParsedMessage(msg)
+        return self.parsed_article
+
+    @property
+    def html(self):
+        content = u"<div id=\"message{}\"><table class=\"messagetable\"><tbody>".format(self.id) +\
+            u"<tr><td><b>Date</b></td><td>{}</td></tr>".format(str(self.created)[:16]) +\
+            u"<tr><td><b>From</b></td><td>{}</td></tr>".format(self.sender_name) +\
+            u"<tr><td><b>Subject</b></td><td>{}</td></tr>".format(self.subject)
+
+        if len(self.article.attachments) > 0:
+            content += u"<tr><td><b>Attachments</b></td><td>"
+            for index, name in enumerate(self.article.attachments):
+                content += u"<a href=\"/{}/attachment/{}.{}\">{}</a>&nbsp;&nbsp;".format(self.mailing_list, self.id, index, name)
+            content += u"</td></tr>"
+
+        content += u"<tr><td></td><td>"
+        if self.article.html:
+            content += self.article.html
+        elif self.article.text:
+            content += u"<pre>{}</pre>".format(self.article.text)
+
+        content += u"</td></tr></tbody></table></div><hr />"
+        return content
+
+    def get_dict(self, with_article):
+        data = {
+            "id": self.id,
+            "sender": self.sender,
+            "messageid": self.messageid,
+            "subject": self.subject,
+            "created": self.created,
+            "parentmessageid": self.parentmessageid
+        }
+        if with_article:
+            data["article"] = self.article
+        return data
+
+
+def get_thread_heads(mailing_list, page=1, page_size=50, year=None, month=None):
+    return db.Message.select().join(db.MailingList).where(
+        (db.Message.parentmessageid.is_null())
+        &(db.MailingList.name == mailing_list)
+    ).order_by(db.Message.created.desc()).paginate(page, page_size)
+
+
+
+
+class Page(object):
+    def __init__(self, mailing_list=None):
+        self.mailing_list = mailing_list
+        self.content = unicode()
+        self.title = unicode()
+        self.search = "<td><a href=\"/{}/search\">Search</a></td>".format(mailing_list) if mailing_list else unicode()
+
+    def create(self, data=None):
+        pass
+
+    def render(self, data=None):
+        self.create(data)
+        with open(u"templates/base.html", "r") as f:
+            base = unicode(f.read())
+        page = base.replace(u"V_CONTENT", self.content).replace(u"V_SEARCH", self.search).replace(u"V_TITLE", self.title)
+        if self.mailing_list:
+            page = page.replace(u"V_MAILINGLIST", self.mailing_list.replace("mailinglist.", ""))
+        print u"Content-Type: text/html; charset=utf-8\r\n\r\n"
+        sys.stdout.write(page.encode("utf-8"))
+        close_nntp()
+        #print page
+
+
+class ListsOverview(Page):
+    def create(self, data=None):
+        self.content = "Select a mailing list from the menu above to view messages."
+        self.title = "Csound mailing lists"
+
+
+class Overview(Page):
+    def create(self, data=None):
+        ml_nicename = "V_MAILINGLIST mailing list archive"
+        page = data.get("page") or 1 if data else 1
+        self.title = u"{}, page {}".format(ml_nicename, page)
+
+        pagination = unicode()
+        if page != 1:
+            pagination += u"<a href=\"/{}/page/{}\">Previous</a>&nbsp;".format(self.mailing_list, page-1)
+        for x in range(max(1, page-10), page+10):
+            if x == page:
+                pagination += u"{} ".format(x)
+            else:
+                pagination += u"<a href=\"/{0}/page/{1}\">{1}</a>&nbsp;".format(self.mailing_list, x)
+        pagination += u"<a href=\"/{}/page/{}\">Next</a>&nbsp;".format(self.mailing_list, page+1)
+
+        threads = get_thread_heads(self.mailing_list, page, 50)
+        content = u"<h2>{}</h2><h3>Page {}</h3>{}".format(ml_nicename, page, pagination) +\
+            u"<table id=\"threads\"><thead><tr><th>Date</th><th>From</th><th>Subject</th></tr></thead><tbody>"
+        odd = True
+        for thread in threads:
+            tmessage = Message(self.mailing_list, db_msg=thread)
+            content += u"<tr class=\"{}\"><td>{}</td><td>{}</td><td><a href=\"/{}/thread/{}\">{}</a></td></tr>".format(
+                "tbl{}".format("odd" if odd else "even"),
+                unicode(tmessage.created)[:16],
+                tmessage.sender_name,
+                self.mailing_list,
+                thread.id,
+                tmessage.subject
+            )
+            odd = not odd
+
+
+        content += u"</tbody></table><div id=\"pagination\">{}</div>".format(pagination)
+        self.content = content
+
+
+class Thread(Page):
+    def create(self, data=None):
+        messages = list()
+        content = unicode()
+        # nasty hack, parameter passing is not working, ugh, but int at least
+        sql = "SELECT id, messageid, subject, sender, created, parentmessageid FROM f_getthread('{}',{});".format(self.mailing_list, int(data["threadid"]))
+        cursor = db._db.execute_sql(sql)
+        for item in cursor:
+            messages.append(Message(self.mailing_list, raw_msg=item))
+        cursor.close()
+        first = True
+        for msg in messages:
+            if first:
+                first = False
+                subject = msg.subject
+                content = u"<h2>{}</h2>".format(subject)
+                self.title = subject
+            content += msg.html
+
+        self.content += content
+
+
+class About(Page):
+    def create(self, data=None):
+        self.title = "About the Csound mailing list archive"
+        with open("templates/about.html", "r") as f:
+            self.content = f.read()
+
+
+class Attachment(object):
+    def __init__(self, mailing_list):
+        self.mailing_list = mailing_list
+    def render(self, data=None):
+        idents = data["ident"].split(".")
+        messageid = db.Message.select().where(db.Message.id == idents[0]).get().messageid
+        result = get_nntp(self.mailing_list).article(messageid)
+        parsed = ParsedMessage("\n".join(result[3]), int(idents[1]))
+        close_nntp()
+        sys.stdout.write(parsed.attachment.encode("utf-8"))
+        
+
+class SearchResult(Page):
+    def create(self, data=None):
+        form = cgi.FieldStorage()
+        term = form.getvalue("term")
+
+        if not term:
+            self.title = "Search the V_MAILINGLIST mailing list archive"
+            with open("templates/search.html", "r") as f:
+                self.content = f.read()
+            return
+
+        title = u"Search results for '{}'".format(term)
+        self.title = title
+        content = u"<h2>{}</h2>".format(title) +\
+            u"<table id=\"threads\"><thead><tr><th>Date</th><th>From</th><th>Subject</th></tr></thead><tbody>"
+
+        if form.getvalue("searchbody"):
+            query = db.Message.select(db.Message).join(db.MailingList).where(
+                (db.MailingList.name == self.mailing_list) 
+                &(
+                    (db.Message.subject.contains(term) if form.getvalue("searchsubject") else True)
+                    | (db.Message.sender.contains(term) if form.getvalue("searchsender") else True)
+                    | (db.MessageText.data.match(term) if form.getvalue("searchbody") else True)
+                )
+            ).join(db.MessageText, join_type=db.JOIN.LEFT_OUTER).distinct().order_by(db.Message.created.desc()).limit(200)
+        else:
+            query = db.Message.select().join(db.MailingList).where(
+                (db.MailingList.name == self.mailing_list) 
+                &(
+                    (db.Message.subject.contains(term) if form.getvalue("searchsubject") else True)
+                    | (db.Message.sender.contains(term) if form.getvalue("searchsender") else True)
+                )
+            ).distinct().order_by(db.Message.created.desc()).limit(200)
+
+        odd = True
+        for message in query:
+            tmessage = Message(self.mailing_list, db_msg=message)
+            content += u"<tr class=\"{0}\"><td>{1}</td><td>{2}</td><td><a href=\"/{5}/thread/{3}#message{3}\">{4}</a></td></tr>".format(
+                "tbl{}".format("odd" if odd else "even"),
+                unicode(tmessage.created)[:16],
+                tmessage.sender_name,
+                message.id,
+                tmessage.subject,
+                self.mailing_list
+            )
+            odd = not odd
+
+        content += u"</tbody></table>"
+        self.content = content
+
+
+
+def handle():
+    path = os.environ.get("REQUEST_URI")
+    if not path or path == "/":
+        return ListsOverview().render()
+
+    parts = path.split("/")[1:]
+
+    if parts[0] == "about":
+        return About().render()
+
+    mailing_list = parts[0]
+
+    if mailing_list not in config.lists:
+        return ListsOverview().render()
+
+
+
+    if len(parts) == 1:
+        return Overview(mailing_list).render()
+    rtype = parts[1]
+
+    if rtype == "thread":
+        return Thread(mailing_list).render({"threadid": int(parts[2])})
+    elif rtype == "page":
+        return Overview(mailing_list).render({"page": int(parts[2])})
+    elif rtype == "search":
+        return SearchResult(mailing_list).render()
+    elif rtype == "attachment":
+        return Attachment(mailing_list).render({"ident": parts[2]})
+
+    return Overview(mailing_list).render()
+
+
+if __name__ == "__main__":
+    try:
+        handle()
+    except Exception, ex:
+        print "Content-Type: text/plain\r\n\r\n" + str(ex)
+        import traceback
+        print "\n\n" + traceback.format_exc()
diff --git a/importer.py b/importer.py
new file mode 100644
index 0000000..c935c1d
--- /dev/null
+++ b/importer.py
@@ -0,0 +1,29 @@
+import sys
+import mailbox
+import config
+import messageparser
+
+
+
+class MboxProcess(object):
+    def __init__(self, filename, newsgroup):
+        msg_number = 0
+        self.newsgroup = newsgroup
+        self.filename = filename
+        self.mbox = mailbox.mbox(filename)
+        for message in self.mbox:
+            self.process_message(message)
+            msg_number += 1
+        print "{} messages".format(msg_number)
+
+
+    def process_message(self, message):
+        post = messageparser.Parser(message)
+        try:
+            post.process()
+        except Exception, ex:
+            print ex
+
+
+if __name__ == "__main__":
+    MboxProcess(sys.argv[1], config.newsgroup)
diff --git a/live_message_import.py b/live_message_import.py
new file mode 100644
index 0000000..e970873
--- /dev/null
+++ b/live_message_import.py
@@ -0,0 +1,30 @@
+import config
+import emailprocessor
+import messageparser
+import db
+
+
+
+def run_lists():
+    for list_name, definition in config.lists.iteritems():
+        if not definition:
+            continue
+        db_mailinglist = db.MailingList.select().where(db.MailingList.name == list_name).get()
+        def process_func(message):
+            post = messageparser.Parser(message, db_mailinglist)
+            try:
+                return post.process()
+            except Exception, ex:
+                return False
+
+        processor = emailprocessor.EmailProcessor(
+            process_func,
+            definition["mail_host"],
+            definition["mail_user"],
+            definition["mail_password"]
+        )
+
+
+if __name__ == "__main__":
+    run_lists()
+    db.housekeeping()
diff --git a/messageparser.py b/messageparser.py
new file mode 100644
index 0000000..1a82de6
--- /dev/null
+++ b/messageparser.py
@@ -0,0 +1,203 @@
+import datetime
+import db
+import config
+import html2text
+import re
+import os
+from dateutil import parser
+from email.utils import parseaddr
+import random
+
+convert_to_x = [
+    "nntp-posting-host",
+    "nntp-posting-date",
+    "injection-info",
+    "received",
+    "expires",
+    "expiry-date"
+]
+
+convert_to_xorig = [
+    "x-trace",
+    "x-complaints-to",
+    "xref"
+]
+
+
+def analyse_line(db_message, index, line):
+    if (len(line.strip())) == 0:
+        return
+
+    parts = line.split(" ")
+    for part in parts:
+        if len(part) > 20:
+            return
+
+    db.MessageText.create(
+        message=db_message,
+        line=index+1,
+        data=db.fn.to_tsvector(line)
+    )
+
+
+def analyse_text(db_message, text):
+    for index, line in enumerate(text.split("\n")):
+        try:
+            analyse_line(db_message, index, line)
+        except Exception, ex:
+            print "line analysis error " + str(ex)
+
+
+class Parser(object):
+    def __init__(self, message, db_mailinglist):
+        self.message_id = None
+        self.message = message
+        self.db_message = None
+        self.parent = None
+        self.db_mailinglist = db_mailinglist
+
+    @property
+    def newsgroup(self):
+        return self.db_mailinglist.name
+
+    def datetime_format(self, dt):
+        return datetime.datetime.strftime(dt, "%a, %d %b %Y %H:%M:%S %z")
+
+
+    def parse_headers(self):
+        headers = dict()
+        fallback_date = None
+        preferred_date = None
+
+        # process headers
+        for key in self.message.keys():
+            lkey = key.lower()
+            converted = False
+
+            if lkey in convert_to_x:
+                headers["X-{}".format(key)] = self.message[key]
+                converted = True
+
+            if lkey in convert_to_xorig:
+                headers["X-Orig-{}".format(key)] = self.message[key]
+                converted = True
+
+            if lkey in ("message-id", "message_id"):
+                match = re.findall("<.*@.*>", self.message[key])
+                if not match:
+                    raise Exception("Invalid message-ID")
+                self.message_id = headers[key] = match[0]
+            elif lkey == "from":
+                if "@" in parseaddr(self.message[key])[1]:
+                    headers[key] = self.message[key]
+                else:
+                    headers["X-{}".format(key)] = self.message[key]
+                    headers[key] = "original_address@invalid.com"
+            elif lkey == "in-reply-to":
+                if self.message[key]:
+                    match = re.findall("<.*@.*>", self.message[key])
+                    if not match:
+                        raise Exception("Invalid In-reply-to")
+                    self.parent = headers[key] = match[0]
+                else:
+                    self.parent = None
+            elif lkey == "nntp-posting-date":
+                fallback_date = parser.parse(self.message[key])
+            elif lkey == "date":
+                try:
+                    preferred_date = parser.parse(self.message[key])
+                except: 
+                    preferred_date = parser.parse(self.message[key][:31])
+            elif lkey == "followup-to":
+                headers[key] = self.newsgroup
+            elif lkey == "newsgroups":
+                pass
+            elif lkey == "path":
+                headers[key] = self.message[key].replace(".POSTED", "").replace("!!", "!")
+            elif not lkey.startswith("x-google-") and not converted:
+                headers[key] = self.message[key]
+
+        if "path" not in headers and "Path" not in headers:
+            headers["Path"] = "bpm!bpm"
+
+        headers["Newsgroups"] = self.newsgroup
+
+        # determine the preferred date to use
+        if not preferred_date and fallback_date:
+            the_date = fallback_date
+        else:
+            the_date = preferred_date
+
+        if not self.message_id:
+            self.message_id = "<{}{}@in.bpm>".format(datetime.datetime.now().strftime("%s%f"), random.random())
+            headers["Message-ID"] = self.message_id
+
+        existing = db.Message.select().where(db.Message.messageid == self.message_id)
+        if existing.exists():
+            r = existing.get()
+            r.created = the_date  # remove after first import, this is a hackathon
+            r.save()
+            #return None    # remove after first import, this is a hackathon
+        else:
+            self.db_message = db.Message.create(
+                messageid=self.message_id,
+                subject=headers["Subject"],
+                sender=headers["From"],
+                parentmessageid=self.parent,
+                mailinglist=self.db_mailinglist,
+                created=the_date
+            )
+
+        # set date
+        headers["Date"] = self.datetime_format(the_date)
+        
+        # format as string and return
+        header_string = str()
+        initial_headers = ["From", "Subject", "Newsgroups", "Date", "Message-ID", "References", "Lines"]
+        for ini in initial_headers:
+            value = headers.get(ini)
+            if value:
+                header_string += "{}: {}\n".format(ini, value.strip())
+        for key, val in headers.iteritems():
+            if key not in initial_headers:
+                header_string += "{}: {}\n".format(key, val.strip())
+        return header_string + "\n"
+
+    def text_only_body(self, message):
+        output = str()
+        for part in message.walk():
+            ct = part.get_content_type()
+            if ct == "text/plain":
+                output += part.get_payload(decode=True)
+            elif ct == "text/html":
+                try:
+                    output += html2text.html2text(part.get_payload(decode=True))
+                except:
+                    pass
+        return output
+
+    def parse_body(self):
+        return self._get_payload(self.message)
+
+    def process(self):
+        headers = self.parse_headers()
+        if headers is None:
+            print "Already exists " + self.message_id
+            return
+
+        parts = str(self.message).split("\n\n")
+        body = "\n\n".join(parts[1:])
+
+        #analyse_text(self.db_message, self.text_only_body(self.message))
+
+        do_post = False
+        try:
+            string_message = headers + body
+            with open("/tmp/msg", "w") as f:
+                f.write(string_message)
+            do_post = True
+        except Exception, ex:
+            print "Message formatting problem: ", ex
+        if do_post:
+            result = os.system("{} /tmp/msg".format(config.rnews))
+            return result == 0
diff --git a/templates/about.html b/templates/about.html
new file mode 100644
index 0000000..d6983c9
--- /dev/null
+++ b/templates/about.html
@@ -0,0 +1,23 @@
+<h2>About the Csound mailing list archives</h2>
+<div>
+    The Csound mailing list archives contains all available messages from the Csound users, dev and tekno mailing lists, and are
+    updated on a regular basis (except tekno, which has been discontinued. 
+<br />
+<br />
+	To post messages, subscribe to the mailing lists at
+	<ul>
+		<li>Users: <a href="https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND">https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND</a></li>
+		<li>Dev: <a href="https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND-DEV">https://listserv.heanet.ie/cgi-bin/wa?A0=CSOUND-DEV</a></li>
+	</ul>
+</div>
+<br />
+<div>
+    This site is maintained by Richard Knight. Any queries/questions/ideas can be addressed through the mailing list
+    itself.
+</div>
+<br />
+<div>
+    Messages are stored in a NNTP server and can be viewed directly with a newsreader by visiting <i>1bpm.net</i>
+    and subscribing to the <i>mailinglist.csound</i> and <i>mailinglist.csound-dev</i>groups.<br />
+    The website provides additional indexing and full-text search capabilities.
+</div>
diff --git a/templates/base.html b/templates/base.html
new file mode 100644
index 0000000..fec4b39
--- /dev/null
+++ b/templates/base.html
@@ -0,0 +1,22 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <title>V_TITLE</title>
+    <link rel="stylesheet" href="/app/base.css" />
+</head>
+<body>
+    <table id="tblmenu" width="100%"><tbody>
+        <tr>
+            <td><a href="/mailinglist.csound">Csound</a></td>
+            <td><a href="/mailinglist.csound-dev">Csound-dev</a></td>
+            <td><a href="/mailinglist.csound-tekno">Csound-tekno</a></td>
+			V_SEARCH
+            <td><a href="/about">About</a></td>
+        </tr>
+    </tbody></table>
+    <div id="content">
+        V_CONTENT
+    </div>
+</body>
+</html>
diff --git a/templates/search.html b/templates/search.html
new file mode 100644
index 0000000..7695f62
--- /dev/null
+++ b/templates/search.html
@@ -0,0 +1,20 @@
+<h2>V_TITLE</h2>
+<div id="search">
+    <form action="/mailinglist.V_MAILINGLIST/search/" method="POST">
+        <table><tbody>
+            <tr>
+                <td>in</td>
+                <td>
+                    <input type="checkbox" name="searchbody" /> body <br />
+                    <input type="checkbox" checked="checked" name="searchsubject" /> subject <br />
+                    <input type="checkbox" checked="checked" name="searchsender" /> sender
+                </td>
+            </tr>
+            <tr>
+                <td>for</td>
+                <td><input type="text" name="term" /></td>
+            </tr>
+        </tbody></table>
+        <input type="submit" value="Search" />
+    </form>
+</div>
diff --git a/tester.py b/tester.py
new file mode 100644
index 0000000..6275a80
--- /dev/null
+++ b/tester.py
@@ -0,0 +1,6 @@
+import email
+import messageparser
+x = open("/tmp/message", "r").read()
+oo = email.message_from_string(x)
+x = messageparser.Parser(oo)
+print x.process()
\ No newline at end of file