krh@65: Towards replacing rpm + yum (0.1):
krh@65: 
krh@214: - drop the filelists from the main package set file, split out to a
krh@214:   secondary file.  for package sets that depend on other package sets,
krh@214:   we need to be able to generate properties with owning packages that
krh@214:   are in another set.  this way, a package that requires a file, will
krh@214:   look up the provides in the set and find the package that owns it
krh@214:   and then try mark that for update.
krh@214: 
krh@82: - installer part:
krh@65: 
krh@82:    - pre install check; check that dirs can be created (no files where
krh@82:      want to create dirs), move config files according to file
krh@82:      flags. (.rpmnew etc)
krh@65: 
krh@82:    - store rpm headers for installed packages.
krh@65: 
krh@214:    - implement rpm uninstall and update.
krh@214: 
krh@214:    - triggers? just say no?
krh@214: 
krh@189: - rpm seems to consider glibc > 2.6.90 to mean greater than
krh@189:   2.6.90-anything.  That is, a comparison that doesn't mention the
krh@189:   release field, shouldn't regard the release field of pkgs it
krh@189:   compares against.  glibc-common-2.6.90 has
krh@189: 
krh@189: 	conflicts: glibc < 2.6.90, glibc > 2.6.90
krh@189: 
krh@189:   since rpm doesn't let you do glibc != 2.6.90, and
krh@189: 
krh@189: 	requires: glibc = 2.6.90
krh@189: 
krh@189:   will always pull in glibc.  But even with a != relation, would
krh@189:   glibc-2.6.90-16 be equal to 2.6.90?  glibc 2.7.90-8 dropped it in
krh@189:   favor of requires = 2.7.90-8 (#225806).
krh@189: 
krh@65: - signed packages
krh@65: 
krh@70: - space calculation before transaction, but ideally, do a number of
krh@70:   smaller transactions.
krh@70: 
krh@70: - pre-link changing binaries and libs on disk screwing up checksum?
krh@70: 
krh@82: - pipelined download and install; topo-sort packages in update set,
krh@82:   pick one with all deps in the current set, add it to the current set
krh@82:   and satisfy deps against update set => result: minimal update
krh@82:   transaction.  Queue download and install/update transaction for the
krh@82:   packages in the minimal set, start over.  This also makes the
krh@82:   installation phase much more interruptible, basically just stop
krh@82:   after a sub-transaction finishes.  As we keep the update set around
krh@82:   as a target, we can restart if needed.  Probably don't need to, can
krh@82:   just do a new update.  During a sub-transaction we should keep the
krh@82:   target set (i.e. the current set to be) around as a lock file
richard@310:   (system.rzdb.lock or so, see git) so that razor updates are
krh@82:   prevented if the systems crashes during an update.
krh@74: 
krh@214: - implement depsolving between multiple package sets by creating an
krh@214:   iterator that has a sorted list of all installed pkgs from all sets,
krh@214:   all installed requires from all sets, all installed provides from
krh@214:   all sets etc.  could be a list of tuples (pkgs index, set index).
krh@214:   should simplify even the two-set depsolving a bit since we can
krh@214:   pretend there's just one set.  this should also be useful for the
krh@214:   'overlay set' idea where the system set is actually made up of a
krh@214:   number of sets, but typically a read-only set from a read-only fs
krh@214:   and a read-write set from a r/w fs.
krh@214: 
krh@214: - locking: we use advisory file locking on the system set
richard@310:   (/var/lib/razor/system.rzdb) to indicate a transaction is in
krh@214:   progress.  The locking algorithm is as follows:
krh@214: 
krh@214:     1. obtain advisory lock on system set.  if this is already taken,
krh@214:        we know that a process is actively modifying the system set and
krh@214:        we have to wait.  there's a fcntl that lets you block for the
krh@214:        lock to go away.
krh@214: 
richard@310:     2. if a system-next.rzdb file already exists an earlier razor
krh@214:        process was interrupted or crashed and we may want to clean
richard@310:        that up.  the system-next.rzdb file will record what the
krh@214:        previous instance was trying to do and we can just replay that
krh@214:        to clean up.
krh@214: 
krh@214:     3. create the new package set whichever way and write it to
richard@310:        system-next.rzdb, then start installing/removing rpms.
krh@214: 
richard@310:     4. When the update is complete, rename system-next.rzdb to
richard@310:        system.rzdb and remove the advisory lock.
krh@214: 
krh@214:   we should probably introduce a new object that encapsulates this
krh@214:   sequence, the filename conventions, rpm cache, e.g. struct
krh@214:   razor_image, with operations such as
krh@214: 
krh@214: 	#define RAZOR_IMAGE_READ	0x01
krh@214: 	#define RAZOR_IMAGE_WRITE	0x02
krh@214: 
krh@214: 	struct razor_image *
krh@214: 	razor_image_open(const char *root, unsigned int flags);
krh@214: 
krh@214: 	int
krh@214: 	razor_image_begin_transaction(struct razor_image *image,
krh@214: 				      struct razor_set *target);
krh@214: 
krh@214: 	int
krh@214: 	razor_image_finish_transaction(struct razor_image *image);
krh@214: 
krh@214:   the transaction pipelineing described above sits on top of this,
krh@214:   since each step there needs to complete a full transaction that
krh@214:   writes out a new package set.
krh@214: 
krh@214:   for overlay package sets we could do something like
krh@214: 
krh@214: 	struct razor_image *
krh@214: 	razor_image_open_with_base(const char *root, unsigned int flags,
krh@214: 				   struct razor_image *base);
krh@214: 
krh@214:   where base specifies the r/o package set it's layered on.  this
krh@214:   allows for stacking several layers of images.
krh@214: 
krh@213: 
krh@213: Package set file format items:
krh@213: 
krh@213: - drop the 4k section alignment
krh@213: 
krh@213: - just use strings for header identifiers, make the string pool
krh@213:   section have a fixed string (maybe make "strings" always the first
krh@213:   string so its index is 0), or maybe just require that it's the first
krh@213:   section in the file.
krh@213: 
richard@310: - nail down byte-order of rzdb file.
krh@213: 
krh@213: - version the sections in the file, put the element size in the header
krh@213:   so we can add stuff to elements in a backwards compatible way.
krh@213:   maybe not necessary, we can just add sections that augment the
krh@213:   sections we want to add to (similar to how rpm has add versioned
krh@213:   deps).
krh@213: 
krh@213: 
krh@65: Misc ideas:
krh@65: 
krh@1: - keep history of installed packages/journal of package transaction,
krh@1:   so we can roll back to yesterday, or see what got installed in the
krh@1:   latest yum update.
krh@1: 
krh@18: - use hash table for package and property lists so we only store
krh@18:   unique lists (like for string pool).
krh@40: 
krh@40: - use existing, running system as repo; eg
krh@40: 
krh@40: 	razor update razor://other-box.local evince
krh@40: 
krh@40:   to pull eg the latest evince and dependencies from another box.  We
krh@40:   should be able to regenerate a rzr pkg from the system so we can
krh@40:   reuse the signature from the originating repo.
krh@41: 
krh@41: - Ok, maybe the fastest package set merge method in the end is to use
krh@41:   the razor_importer, but use a hash table for the properties.  This
krh@41:   way we can assign them unique IDs immediately (like tokenizing
krh@41:   strings).
krh@47: 
richard@310: - test suite should be easy, just keep .rzdb files around and test
krh@47:   different type of upgrades that way (obsoletes, conflicts, file
krh@47:   conflicts, file/dir problems etc).  Or maybe just keep a simple file
richard@310:   format ad use a custom importer to create the .rzdb files.
krh@47: 
krh@63: - overlay package sets?  mount a read-only /usr over nfs or from the
krh@63:   virt-host and have a local package set overlaid over the read-only
krh@63:   one.  shouldn't need new features in the core package set data
krh@63:   structure, but should be just conventions on top.  we have the base
krh@63:   package set from the r/o system, the overlay set from the local
krh@63:   system and we can have an effective package set which is the merge
krh@63:   of everything from the overlay into the base set.  the effective set
krh@63:   is easy to compute and we could do it on the fly or cache it.  or
krh@63:   maybe the effective set is the on-disk representation and the
krh@63:   overlay can be computed when needed, we just keep a link back to the
krh@63:   base.
krh@71: 
krh@71: - incremental rawhide repo updates? instead of downloading 10MB zipped
richard@310:   repo every time, download a diff rzdb?  Should be pretty small,
krh@82:   especially if we don't have file checksums in metadata.  Filenames
krh@82:   and properties are for the most part already present, typically just
krh@82:   a version bump plus maybe tweaking a couple requires.  The upstream
krh@82:   repo can store multiple incremental updates in one big file and
krh@82:   provide an index file that maps updates for a given date (we should
krh@82:   use repo-file checksums though) to a range in the file: Download the
richard@310:   index file, search for a match for your latest rawhide.rzdb file,
krh@82:   download range of updates that brings it up to date.
krh@74: 
krh@74: - use hash tables for dirs when importing files to avoid qsorting all
krh@74:   files in rawhide.
krh@75: 
krh@82: Bugs:
krh@82: 
krh@82: - eliminate duplicate entries in package property lists.