Another milestone on the road to Zen with Ingeniweb components
Latest version : 2.6.1

File System Storage

> FileSystemStorage (FSS) is an Archetypes storage for storing fields raw values on the file system. This storage is used to avoid unnecessary growth of the ZODB's FileStorage (Data.fs) when using a lot of large files. <

About File System Storage

FileSystemStorage (FSS) is an Archetypes storage for storing fields raw values on the file system. This storage is used to avoid unnecessary growth of the ZODB's FileStorage (Data.fs) when using a lot of large files.

File System Storage has been designed by the INGENIWEB team.

File System Storage 2.6.1 is licensed under the GNU GPL license.

Do you like this product? You can support its authors by clicking the button below:
Support This Project

Screenshots

Here comes a few screenshots of FileSystemStorage.

doc/fss_item_view.png

View of content using FileSystemStorage with ImageField, TextField and FileField

doc/fss_management_configlet.png

Manage your storage paths in configlet

doc/fss_maintenance_configlet.png

View of maintenance screen. It will help to use FileSystemStorage

Readme

FileSystemStorage

By Ingeniweb.



1   About FileSystemStorage

FileSystemStorage (FSS) is an Archetypes storage for storing fields raw values on the file system. This storage is used to avoid unnecessary growth of the ZODB's FileStorage (Data.fs) when using a lot of large files.

Please note that FSS is a Plone component for content types developers. Do not expect anything more in your Plone site if you don't use Plone components that may use FSS, such as AttachmentField.

2   Copyright and license

Copyright (c) 2005 - 2007 Ingeniweb SAS

This software is subject to the provisions of the GNU General Public License, Version 2.0 (GPL). A copy of the GPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE

See the LICENSE file that comes with this product.

3   Requirements

4   Migration

4.1   From FSS 2.5.x and older

The way we configure the storage and backup paths as well as the storage strategies HAVE CHANGED IN DEPTH. There's no automatic migration from this version. Please create a configuration file that clones your old configlet preferences as indicated in FSS main configuration.

5   Installation

  • Put FileSystemStorage product in your zope instance Products directory.
  • Remove configure.zcml if you're running a Zope version older than 2.9.6.
  • Read carefully Storage strategies and FSS main configuration below and configure your instance in plone-filesystemstorage.conf accordingly.
  • Start your Zope instance
  • In ZMI, go into your plone site and install the product using portal_quickinstaller
  • Optionally, tweak some - self explained - preferences in the FileSystemStorage configuration panel of your Plone site.

6   Storage strategies

A storage strategy defines how field values will be stored in your files system.

6.1   Strategies overview

FSS comes with 4 storage strategies. Each strategy requires two directories:

  • The storage directory stores the values of relevant fields in files according to the selected strategy.
  • The backup directory that keeps the values of the fields of deleted contents for "Undo" purpose.
  • Only one storage strategy can be selected on a given Plone site, but different Plone site in the same Zope instance may have distinct storage strategies.
  • You need the best performances: choose the Flat storage strategy or Directory storage strategy.
  • You need a storage that looks like your Plone site (in example to publish the storage directory in a read only samba share): choose the Site storage strategy 1 or the Site storage strategy 2.

See FSS main configuration to configure the best suited strategy on your Zope instance and Plone sites.

6.2   Flat storage strategy

All field values are stored in a flat structure. This strategy is the default one.

  • Filename of these stored values: <uid of content>_<field name>
  • Filename of rdf file: <uid of content>_<field name>.rdf
  • Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated.

Example of storage:

fssDirectory
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

6.3   Directory storage strategy

All field values are stored in a directory structure. Sub directories are defined on two level. First level of directory uses the 2 characters of content uid. Second level of directory uses the 4 characters of content uid. Backup files are stored in a flat structure.

  • Filename of these stored values: <uid of content>_<field name>
  • Filename of rdf file: <uid of content>_<field name>.rdf
  • Filename of backup values: <uid of content>_<field name>.bak

RDF files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- f42
   |
   |- f42ad
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

Depending in one hand on your file system performances facing a huge amount of files in the same directory, and in the other hand on the number of contents relying on FSS, you might choose this strategy or the Flat storage strategy.

6.4   Site storage strategy 1

All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

  • Filename of these stored values: Filename of field value or field name if not defined
  • Filename of rdf file: <field name>.rdf
  • Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- image
         |  |
         |  |- moutain.jpg
         |  |
         |  |- image.rdf
         |
         |- file
            |
            |- diary.odt
            |
            |- file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

6.5   Site storage strategy 2

All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

  • Filename of these stored values: Filename of field value or field name if not defined
  • Filename of rdf file: <field filename>.rdf
  • Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- fss.cfg
         |
         |- moutain.jpg
         |
         |- mountain.jpg.rdf
         |
         |- diary.odt
         |
         |- diary.odt.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

7   FSS main configuration

FSS is mainly configured the ZConfig way. At startup, the configuration file will be searched (in that order) in:

  • $INSTANCE_HOME/etc/plone-filesystemstorage.conf
  • /path/to/FileSystemStorage/etc/plone-filesystemstorage.conf
  • /path/to/FileSystemStorage/etc/plone-filesystemstorage.conf.in

A sample working configuration in provided in this last file. It assumes you have $INSTANCE_HOME/var/fss_storage and $INSTANCE_HOME/var/fss_backup directories, both being read/write enabled to the user that runs the Zope process, unless Zope will raise an error at startup.

All configuration doc you need is in plone-filesystemstorage.conf.in comments.

Inconsistent configuration features raise explicit error messages at Zope startup.

Note that we didn't include this in zope.conf in order to keep Zope 2.7 and Zope 2.8 compatibility.

8   Configuration panel

8.1   FSS and RDF

As mentioned above in Storage strategies, FSS can optionally store RDF files in the back-end storage. These RDF files conform the DCMES-XML standard on XML expression of DublinCore elements.

If you select the Flat storage strategy, the RDF files may be used to build a files tree as close as possible of the Plone tree structure from the storage back-end, with the build_fs_tree.py utility:

$ cd /path/to/FileSystemStorage/bin
$ python build_fs_tree.py --help

8.2   FSS maintenance

The "Maintenance" tab of FSS configuration panel shows some statistics about the amount of files managed by FSS.

In addition 2 additional buttons are provided:

  • Update FSS files : Cleans up the back-end storage directories. Files not referenced from a content object are removed.
  • Update RDF files : Builds all RDF files, in the event you chose to select the generetion of RDF files after having inserted relevant contents.

9   Zope backups

Of course, you need to synch your Data.fs backups with the storage paths backups, unless you may restore corrupted/incomplete sites.

10   Caveats and pittfalls

Note that whatever's the selected strategies, DO NEVER CHANGE ANY FILE IN THE STORAGE PATHS UNLESS YOU KNOW WHAT YOU DO, otherwise you'll loose your contents, your job, your money, your friends, your wife/husband and children.

DO NEVER CHANGE THE STRATEGY FOR A PLONE SITE ONCE THIS SITE HAS FIELDS STORED IN ITS FSS STORAGE PATH unless you're loose your contents, your job, ... FSS has no utility to convert a storage format to another one. Sponsorship is welcome - as indicated in Support and feedback - if you need such a feature.

Thought it is possible to share the same storage path and backup path within various Plone sites, since you don't mind of your data in development sites, you should really configure FSS such each Plone site has its own private storage path and backup path. In cas you ignore this warning, DO NEVER CLICK ANY BUTTON IN THE MAINTENANCE TAB OF THE FSS CONFIGURATION PANEL otherwise you'll loose your contents, your job, ...

Zexp exports don't embed the associated FSS storage. So don't move your Plone site within Zope instances using zexp exports unless you move the storage and backup directories along with the zexp file.

After changing your configuration (see FSS main configuration), always restart your Zope instance in foreground since configuration errors are not reported when Zope is started as daemon or Windows service.

11   Developer's corner

11.1   Using FSS in your content types

We assume that creating AT based content types is a familiar activity.

When creating the schema of a content type, you usually don't mention the storage of your fields since AT provides a default ZODB storage. You can choose another storage, either another one that ships in the Archetypes bundle, or a third party storage like FSS.

Here is a small example that doesn't need much comments:

# Usual Zope/CMF/Plone/Archetypes imports
...
from Products.FileSystemStorage.FileSystemStorage import FileSystemStorage
...
my_schema = Schema((
    FileField('file',
              ...
              storage=FileSystemStorage(),
              widget=FileWidget(...)
              ),
    ...
    ))
 ...

You may have a look at examples/FSSItem.py to see a demo content type using FSS. If you want to play with this content type:

$ $INSTANCE_HOME/bin/zopectl stop
$ export FSS_INSTALL_EXAMPLE_TYPES=1
$ $INSTANCE_HOME/bin/zopectl start

11.2   Monkey patching a third party content type

This simple example shows how to plug the ATFile standard type from any custom product. Let's say we're in the __init__.py of your custom product:

...
from Products.FileSystemStorage.FileSystemStorage import FileSystemStorage
from Products.ATContentTypes import atct
...
field = atct.ATFile.schema['file']
field.storage = FileSystemStorage()
field.registerLayer('storage', field.storage)
...

11.3   Customize RDF info set

You can add data to the RDF info set via a FSS dedicated - and optional - hook.

Open the FSS configuration panel and give the name of your personal RDF data script, let's call it fss_custom_rdf for this example. This script will add or change the RDF default data set.

11.3.1   RDF script interface

The script may be in any Plone layer like custom or any other in the skins path.

Such script is expected to have 4 parameters:

  • name: the name of the field
  • instance: the content object being processed
  • properties: FSS field properties mapping, depending on the type of field associated to FSS storage.
  • default_rdf: a mapping structure of the RDF data, including the namespaces, and the elements.

Such script is expected to return an updated default_rdf mapping structure.

Please read the source of classes FSSInfo, FSSFileInfo and FSSImageInfo from module FileSystemStorage.py, RDFWriter from module rdf.py for detailed information on formats of expected data.

11.3.2   Simple example

We need to add the date of RDF file generation in an element like this:

<fss:rdf_date>2007-05-13 13:21:00</fss:rdf_date>

Here's the script:

## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
"""We add the date of RDF file generation"""

rdf_args = default_rdf.copy()

# Add the date of RDF generation
new_props = []
new_props.extend(rdf_args['field_props'])
new_props.append({'id': 'fss:rdf_date', 'value': DateTime().ISO()})
rdf_args['field_props'] = new_props

return rdf_args

11.3.3   More complex example

In this more complex (and stupid - don't do this ;) example, we remove the elements from fss namespace and simulate the standard behaviour for other elements (dc and rdf elements).

Here's the code:

## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
from Products.CMFCore.utils import getToolByName

rdf_args = {}

# Set RDF charset
ptool = getToolByName(instance, 'portal_properties')
rdf_args['charset'] = ptool.site_properties.default_charset

# Set RDF namespaces
rdf_args['namespaces'] = (
    {'id': 'xmlns:rdf', 'value': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'},
    {'id': 'xmlns:dc', 'value': 'http://purl.org/dc/elements/1.1/'},
    {'id': 'xmlns:fss', 'value': 'http://namespace.ingeniweb.com/fss'},)

# Set field url
utool = getToolByName(instance, 'portal_url')
portal_path = utool.getPortalObject().getPhysicalPath()
portal_path_len = len(portal_path)
rel_path = '/'.join(instance.getPhysicalPath()[portal_path_len:])
fss_path = '%s/fss_get/%s' % (rel_path, name)
rdf_args['field_url'] = fss_path

# Set RDF properties
props = (
   {'id': 'dc:title', 'value': instance.title_or_id()},
   {'id': 'dc:description', 'value': instance.Description()},
   {'id': 'dc:language', 'value': instance.Language()},
   {'id': 'dc:creator', 'value': instance.Creator()},
   {'id': 'dc:date', 'value': instance.modified()},
   {'id': 'dc:format', 'value': properties.get('mimetype', 'text/plain')},
   )
rdf_args['field_props'] = props

# Returns the RDF structure
return rdf_args

11.3.4   Recommanded practices on custom RDF

Do not change the existing elements under fss namespace unless the bin/build_fs_tree.py utility won't work. In other terms, do not use the script of the More complex example.

Adding or changing elements to the dc and rdf namespaces may screw up external utilities based on canonical DCMES-XML standard.

Consider adding your own namespace for your custom extra elements.

12   Testing

Please read ./tests/README.txt.

13   Other documentation

See FileSystemStorage/doc.

14   Downloads

You may find newer stable versions of FSS and pointers to related informations (tracker, doc, ...) from http://plone.org/products/filesystemstorage

15   Subversion repository

Stay in tune with the freshest (maybe unstable) versions or participate to the FileSystemStorage evolutions:

https://svn.plone.org/svn/collective/FileSystemStorage

16   Support and feedback

Please read all the documentation that comes with this product before asking for support, unless you might get a RTFM reply ;)

Localisation issues - other than french - should be reported to the relevant translators (see Credits below).

Report bugs using the tracker (the Tracker link from http://plone.org/products/filesystemstorage). Please provide in your bug report:

  • Your configuration (Operating system+Zope+Plone+Products/versions).
  • The storage strategy in use.
  • The full traceback if available.
  • One or more scenario that triggers the bug.

Note that we do not support bug reports on Subversion trunk or branches checkouts.

Mail to Ingeniweb support in English or French to ask for specific support.

Donations are welcome for new features requests

17   Credits


Download

Released versions

Released versions of File System Storage are available here. The current version is 2.6.1.

CVS version

The repository contains the up-to-date versions of our source code. In order to get the HEAD branch of FileSystemStorage, use :

svn co http://svn.plone.org/svn/collective/FileSystemStorage/trunk
          

You can also browse the CVS with your browser.

Documentation

Readme

Please take time the read the Readme

F.A.Q

FAQ

What are the differences between FileSystemStorage and ExternalStorage?

On copy/cut actions, FileSystemStorage doesn't use memory : External storage stores files in a volatile var for copy/cut actions.

Is it easy to use FileSystemStorage?

Just add in your Field definition

storage = FileSystemStorage()

Just take a look at FSSItem example.

Warning: Never use:

myschema['myfield'].storage = FileSystemStorage()

This example never initializes storage layers, thus won't work.

Does large file use StreamIterator?

Yes of course. We save memory at file download.

Can I use the same storage path for many Plone instances?

Don't do this. FileSystemStorage maintenance doesn't work if many Plone instances use the same storage path. See README.txt and the configuration file to configure a couple of storage/backup paths per Plone site.

Which AT Fields can I use with FileSystemStorage?

FileSystemStorage works for:

  • FileField
  • AttachmentField
  • ImageField
  • TextField
  • StringField

Using FileSystemStorage with other fields is useless.

Can I modify content directly on FileSystem?

No it's dangerous, contents using FileSystemStorage are not notified about filesystem modifications. Anyway, you may give a read-only access to the storage path when unsing "Site 1" or "Site 2" strategy, through Samba for example.

Does undo action works ?

Yes, only for deletion. If you delete a content and undo its deletion you can recover data.

Does it works with multiple ZEO clients?

Yes, you need to set storage and backup paths in config file to NFS/Samba/Windows shares.

I have a problem under Plone 2.0.x while copying/pasting a folder containing objects using FSS

Why folder items are not copied ?

PloneFolder copy/paste action doesn't work correctly with AT objects. Use ATCT folders to avoid copy/paste bugs.

Links

History

File System Storage changes

CHANGES

TODO

  • Run unit tests with Zope 2.7 and Plone 2.0
  • Add checksum to be sure files are not corrupted on filesystem
  • Include Czek translation expected from Radim
  • Fix screenshot (deprecated from 2.6)

2.6.1 - 2007/10/29

  • ZConfig bug that put all config sting in lowercase: we test path in lowercase too (ED)
  • Remove the OSError when moving file into backup
  • Plone 3.0 compatibility (GL)
  • Do not override Content-Disposition HTTP header if already set in VirtualBinary.index_html (zegor)
  • filestream_iterator is not "range proof". Add a new range_filestream_iterator to negociate Range request

2.6.0 RC1 - 2007/06/15

  • Changed in depth the way we configure FSS with the ZConfig way. Changing the storage and backup path, and the storage strategy TTW is disabled (GL)
  • As a consequence, changing the storage strategy doesn't require to hack config.py any more (GL)
  • As a consequence, each Plone site in a Zope instance can have its own storage strategy (GL)
  • Configuration view and README.txt view in ZMI (GL)
  • Fixed documentation issues (GL)
  • SiteStorageStrategy and SiteStorageStrategy2 didn't work under Win32 (GL)
  • Make FSS compatible with Plone 2.0 : just rewrite some testcases

2.5.6 - 2007/06/15

  • When renaming a folder containing FSS content, FSSInfo attribute was stolen one each content of the folder. Provides a solution to avoid this problem (CL)

2.5.5 - 2007/06/13

  • Make FSS works when calling _setUID method on an AT object (CL)
  • Add a strategy mark in the storage in a '.strategy' file such we could forbid strategy changes of storages at Zope startup. (i.e., using a 'flat' storage strategy in a storage filled using a 'directory' strategy)

2.5.4 - 2007/03/13

  • Fix a bug on Plone 2.5 when moving a folder containing items using FSS storage. Those items were losing their FSS content. (CL)

2.5.3 - 2007/02/12

  • Don't monkey patch BaseObject __bobo_traverse__ under Plone 2.5, but register ITraversable adapter for this. This change is required as of Archetypes 1.4.1 (see AT HISTORY.txt, search '__bobo_traverse__')

2.5.2 - 2006/10/27

  • Fix bug on maintenance tab: update files and backup, and update RDF (ED)
  • Fix some renaming bugs (ED)
  • Extend index_html method of VirtualFile and VirtualImage to use a filestream iterator. FSS is now compliant with CacheFu
  • Fix bug in FSSTool:updateFSS - replace restoreBackup method by restoreValueFile
  • Add new strategy: SiteStorageStrategy2

2.5.1 - 2006/07/28

  • Check existence of backup and storage paths in configlet
  • Fix bug on maintenance tab

2.5 - 2006/07/19

  • Update tests to use all strategies
  • Add storage strategies : FlatStorageStrategy, DirectoryStorageStrategy, SiteStorageStrategy
  • Patch BaseObject to make it possible to access images like this mysite/myobj/image or mysite/myobj/image_mini
  • Added a docstring on VirtualData class and subclasses. This should fix http://plone.org/products/atphoto/issues/28 : VirtualImage can be published directly by ATImage

2.5RC1

  • Add a hook to customize RDF files
  • Add script to build filesystem tree from FSS files
  • Add RDF implementation

2.1RC2

  • Remove use of ComputedAttribute to avoid strange errors
  • If file is empty don't create an empty file on filesystem

2.1RC1

  • Add Content-Disposition header in fss_get python script
  • Make FSS compliant with AT 1.3.4 and Plone 2.1

2.0RC3

  • Fix bug on delete action (In previous version, it was impossible to delete non required field value)

2.0RC2

  • Add translations
  • Fix bug in configlet when no files are stored on filesystem.

2.0RC1

  • First public release

File System Storage ChangeLog is also available for detailed informations.

Contact