trac-upgrade-and-migration

content history

12y, 281d ago [edited]

Trac Upgrade and Migration

We need to perform the following tasks:

Migrate Trac project to a new host
Upgrade from Trac==0.12.2 to Trac==1.0.1
Migrate tickets from old Trac project to new Trac project

Possible solutions to migrate tickets from old Trac project to new Trac project:

Manual ticket migration
- allows for intelligent migration of tickets
- allows for important ticket proceedures to be properly documented in sphinx
- time consuming
Export old, import new
- moves ALL data to new project (even garbage)
- ticket numbers persist
- NOT time consuming
Research and develop a 'clone to new system' plugin
- allows for intelligent migration of tickets
- allows for important tasks to be documented in Ops Handbook
- time consuming (plugin development + manual migration)
Put old Trac into read-only, leave it running till it dies?
- manual migration of open tickets
- simple as: chmod 000 pass_file
- operation headache, confusing to have two tracs

setup development env from hot-copy of prod

create hot-copy of trac:

trac-admin /path/to/projenv hotcopy /path/to/backupdir

create a tar archive of hot backup:

tar -czvf trac-2013-07-2.tar.gz /cars/archive/trac-hot-copy-backup

on your workstation scp tar archive:

cd ~
mkdir trac
cd trac
scp old-server:/cars/archive/trac-2013-07-2.tar.gz .

unarchive:
```
tar xzvf trac-2013-07-2.tar.gz
```

create virtualenv:

cd trac-hot-copy-backup
virtualenv .env
source .env/bin/activate

install Genshi==0.6 because Trac==0.12.2 breaks with Genshi==0.7:
```
pip install Genshi==0.6
```
install Trac==0.12.2:
```
pip install Trac==0.12.2
```

test setup:

tracd --port=8800 --env-parent-dir ~/trac/

open browser to http://127.0.0.1:8800

upgrade from Trac==0.12.2 to Trac==1.0.0

bring hot-copy tracd server offline

Update the Trac Code to Trac==1.0:

# optional, but these directories are no longer used 
#rm -rf ~/trac/trac-hot-copy-backup/htdocs/ ~/trac/trac-hot-copy-backup/templates/
pip install Trac==1.0

Upgrade the Trac environment:

trac-admin ~/trac/trac-hot-copy-backup upgrade
trac-admin ~/trac/trac-hot-copy-backup wiki upgrade

test setup:

tracd --port=8800 --env-parent-dir ~/trac/

upgrade again from Trac==1.0.0 to Trac==1.0.1

bring hot-copy tracd server offline

Update the Trac Code to Trac==1.0.1:

# optional, but these directories are no longer used 
#rm -rf ~/trac/trac-hot-copy-backup/htdocs/ ~/trac/trac-hot-copy-backup/templates/
pip install Trac==1.0.1

Upgrade the Trac environment:

trac-admin ~/trac/trac-hot-copy-backup upgrade
trac-admin ~/trac/trac-hot-copy-backup wiki upgrade

test setup:

tracd --port=8800 --env-parent-dir ~/trac/

dump ticket tables from hot-copy and import to new project

move to db directory:
```
cd ~/trac/trac-hot-copy-backup/db
```

dump each table related to tickets:

sqlite3 trac.db ".dump ticket" > trac_ticket.sql
sqlite3 trac.db ".dump ticket_change" > trac_ticket_change.sql
sqlite3 trac.db ".dump ticket_custom" > trac_ticket_custom.sql
sqlite3 trac.db ".dump ticket_template_store" > trac_ticket_template_store.sql
sqlite3 trac.db ".dump attachment" > trac_attachment.sql
sqlite3 trac.db ".dump milestone" > trac_milestone.sql
sqlite3 trac.db ".dump pastes" > trac_pastes.sql

copy *.sql to the new trac project db directory
become trac user:
```
sudo su - trac
bash
```

import the tables:

cat trac_ticket.sql | sqlite3 trac.db
cat trac_ticket_change.sql | sqlite3 trac.db
cat trac_ticket_custom.sql | sqlite3 trac.db
cat trac_ticket_template_store.sql | sqlite3 trac.db
cat trac_attachment.sql | sqlite3 trac.db
cat trac_milestone.sql | sqlite3 trac.db
cat trac_pastes.sql | sqlite3 trac.db

copy files/attachments directory from hot-copy to new Trac project. Make sure owner:group is trac:trac:
```
chown -R trac:trac files/attachments
```

trac database tables and explanations

attachment: KEEP - meta data for attachment files. Also need underlying attachment directory.
auth_cookie: DROP - cookie session data for logged in users
cache: DROP - only record: 901198563trac.wiki.api.WikiSystem.pages
component: DROP - we plan to create new components via config management
enum: DROP - we plan to create new enums (priority, severity, ticket_type, resolution) via config management
milestone: KEEP - we have 7 records that need to be moved, Trac defaults to Milestone 1, 2, 3, 4
node_change: DROP - empty
pastes: KEEP - we need to install paste plugin
permission: DROP - we don't need to migrate old permissions. we need a plan to place in config management
report: DROP - we don't need old reports. we might want to back them up as examples
repository: DROP - empty
revision: DROP - empty
session: DROP - session data for when people logged in and what they did to the trac system
session_attribute: DROP - session data for when people logged in and what they did to the trac system
subtickets: KEEP but DROP - we might need to transform this data to work with new subticket plugin
system: DROP - version numbers of plugins and modules
ticket: KEEP - tickets
ticket_change: KEEP - ticket history
ticket_custom: DROP - ticket custom fields, lots of garbage from old plugins. We will be generating a migration from subticket to new childticket plugin that will insert into this table.
ticket_template_store: KEEP - ticket templates which auto fill body message with text, discuss this
version: DROP - empty
wiki: DROP - discuss, we could migrate everything or we could migrate some pages

subtickets migration to child ticket plugin

dump out the data and edit with vim

sqlite3 trac.db
sqlite> .mode csv
sqlite> .out trac_subtickets.csv
sqlite> SELECT child,parent FROM subtickets;
sqlite> .exit
vim trac_subtickets.csv

run the following search and replace transformations:

%s/,/,"parent","#/g
%s/$/");/g
%s/^/INSERT INTO "ticket_custom" VALUES(/g

rename

cp trac_subtickets.csv trac_ticket_children.sql

Trac Upgrade and Migration
#############################

.. contents::

We need to perform the following tasks:

* Migrate Trac project to a new host
* Upgrade from Trac==0.12.2 to Trac==1.0.1
* Migrate tickets from old Trac project to new Trac project

Possible solutions to migrate tickets from old Trac project to new Trac project:

* Manual ticket migration

- allows for intelligent migration of tickets
  - allows for important ticket proceedures to be properly documented in sphinx
  - time consuming

* Export old, import new

- moves ALL data to new project (even garbage)
  - ticket numbers persist 
  - NOT time consuming

* Research and develop a 'clone to new system' plugin

- allows for intelligent migration of tickets
  - allows for important tasks to be documented in Ops Handbook
  - time consuming (plugin development + manual migration)

* Put old Trac into read-only, leave it running till it dies?

- manual migration of open tickets  
  - simple as: chmod 000 pass_file
  - operation headache, confusing to have two tracs

setup development env from hot-copy of prod
===============================================

#. create hot-copy of trac:

.. code-block:: bash

trac-admin /path/to/projenv hotcopy /path/to/backupdir

#. create a tar archive of hot backup:

.. code-block:: bash

tar -czvf trac-2013-07-2.tar.gz /cars/archive/trac-hot-copy-backup

#. on your workstation scp tar archive:

.. code-block:: bash

cd ~
    mkdir trac
    cd trac
    scp old-server:/cars/archive/trac-2013-07-2.tar.gz .

#. unarchive:

.. code-block:: bash

tar xzvf trac-2013-07-2.tar.gz

#. create virtualenv:

.. code-block:: bash
   
    cd trac-hot-copy-backup
    virtualenv .env
    source .env/bin/activate

#. install Genshi==0.6 because Trac==0.12.2 breaks with Genshi==0.7:

.. code-block:: bash

pip install Genshi==0.6

#. install Trac==0.12.2:

.. code-block:: bash

pip install Trac==0.12.2

#. test setup:

.. code-block:: bash

tracd --port=8800 --env-parent-dir ~/trac/
    
#. open browser to http://127.0.0.1:8800

upgrade from Trac==0.12.2 to Trac==1.0.0
===============================================

#. bring hot-copy tracd server offline

#. Update the Trac Code to Trac==1.0:

.. code-block:: bash
    
    # optional, but these directories are no longer used 
    #rm -rf ~/trac/trac-hot-copy-backup/htdocs/ ~/trac/trac-hot-copy-backup/templates/
    pip install Trac==1.0

#. Upgrade the Trac environment:

.. code-block:: bash

trac-admin ~/trac/trac-hot-copy-backup upgrade
    trac-admin ~/trac/trac-hot-copy-backup wiki upgrade

#. test setup:

.. code-block:: bash

tracd --port=8800 --env-parent-dir ~/trac/

upgrade again from Trac==1.0.0 to Trac==1.0.1
===============================================

#. bring hot-copy tracd server offline

#. Update the Trac Code to Trac==1.0.1:

.. code-block:: bash

# optional, but these directories are no longer used 
    #rm -rf ~/trac/trac-hot-copy-backup/htdocs/ ~/trac/trac-hot-copy-backup/templates/
    pip install Trac==1.0.1

#. Upgrade the Trac environment:

.. code-block:: bash

trac-admin ~/trac/trac-hot-copy-backup upgrade
    trac-admin ~/trac/trac-hot-copy-backup wiki upgrade

#. test setup:

.. code-block:: bash

tracd --port=8800 --env-parent-dir ~/trac/

dump ticket tables from hot-copy and import to new project
====================================================================

#. move to db directory:

.. code-block:: bash

cd ~/trac/trac-hot-copy-backup/db

#. dump each table related to tickets:

.. code-block:: bash

sqlite3 trac.db ".dump ticket" > trac_ticket.sql
    sqlite3 trac.db ".dump ticket_change" > trac_ticket_change.sql
    sqlite3 trac.db ".dump ticket_custom" > trac_ticket_custom.sql
    sqlite3 trac.db ".dump ticket_template_store" > trac_ticket_template_store.sql
    sqlite3 trac.db ".dump attachment" > trac_attachment.sql
    sqlite3 trac.db ".dump milestone" > trac_milestone.sql
    sqlite3 trac.db ".dump pastes" > trac_pastes.sql

#. copy \*.sql to the new trac project db directory

#. become trac user:

.. code-block:: bash

sudo su - trac
    bash

#. import the tables:

.. code-block:: bash

#. copy files/attachments directory from hot-copy to new Trac project.  Make sure owner:group is trac:trac:

.. code-block:: bash
   
    chown -R trac:trac files/attachments

trac database tables and explanations
===============================================

attachment
 KEEP - meta data for attachment files.  Also need underlying attachment directory.

auth_cookie
 DROP - cookie session data for logged in users

cache
 DROP - only record: 901198563|1|trac.wiki.api.WikiSystem.pages 
 
component
 DROP - we plan to create new components via config management

enum
 DROP - we plan to create new enums (priority, severity, ticket_type, resolution) via config management

milestone
 KEEP -  we have 7 records that need to be moved, Trac defaults to Milestone 1, 2, 3, 4

node_change
 DROP - empty

pastes
 KEEP - we need to install paste plugin

permission
 DROP - we don't need to migrate old permissions. we need a plan to place in config management

report
 DROP - we don't need old reports. we might want to back them up as examples

repository
 DROP - empty

revision
 DROP - empty

session
 DROP - session data for when people logged in and what they did to the trac system

session_attribute
 DROP - session data for when people logged in and what they did to the trac system

subtickets
 KEEP but DROP - we might need to transform this data to work with new subticket plugin

system
 DROP - version numbers of plugins and modules

ticket
 KEEP - tickets

ticket_change
 KEEP - ticket history

ticket_custom
 DROP - ticket custom fields, lots of garbage from old plugins. We will be generating a migration from subticket to new childticket plugin that will insert into this table.

ticket_template_store
 KEEP - ticket templates which auto fill body message with text, discuss this

version
 DROP - empty

wiki
 DROP - discuss, we could migrate everything or we could migrate some pages

subtickets migration to child ticket plugin
==================================================

dump out the data and edit with vim

.. code-block:: bash

sqlite3 trac.db
 sqlite> .mode csv
 sqlite> .out trac_subtickets.csv
 sqlite> SELECT child,parent FROM subtickets;
 sqlite> .exit
 vim trac_subtickets.csv

run the following search and replace transformations:

.. code-block:: vim

%s/,/,"parent","#/g
 %s/$/");/g
 %s/^/INSERT INTO "ticket_custom" VALUES(/g

rename

.. code-block:: bash

cp trac_subtickets.csv trac_ticket_children.sql

Comments

Leave first comment to start a conversation!