Extracting the models

ego533

New member
Joined
May 10, 2011
Messages
91
 
   Does anyone have the old posts from the Planet Black and White wiki on the model format?

   Extracting the models is evidently possible: [ http://forum.xentax.com/viewtopic.php?f=16&t=2634&hilit=black+and+white ]. The last post was a couple months ago. I think it is a whole different set of work to extract the bones, UV, etc..

   I think this has been brought up before, but I can't find it anymore. Is this correct? : For the creature, it appears that the morph models are in the .csk files, the hair and hair model is in the .cha file. The .cbn file contains attributes of the creature (references to the sounds in the .lug files, animations apparently in an "lhq" format, etc.., does this contain the skeleton and bones?). The .css file contains models of the main body mesh (Or does this contain the bones?)
   Using a hex editor and doing some stupid manual parsing, some of the files can be edited in a slightly useful way.. I always hated the derpy look of the 'Good' wolf; I think 'good' morph can basically just be removed to leave the regular morph in place.. just have the skin change.

 

   Is there any relation of Black and White model formats to Fable? There doesn't appear to be a useful link to either Fable or Fable 2..

-ego533
 
 
  The wayback machine was helpful! [ http://web.archive.org/web/20090413044933/http://wiki.planetblackandwhite.gamespy.com/PBnWModdingWIKI/index.php/Main_Page ]

  Model file information is attached. The files are actually .rtf renamed to .doc. Should open in any Rich Text editor.

-ego533
 
  ... And basically giving it a straight shot, here is what may be the model morph for "Thin Wolf". If someone wants to try to get it in a format to import into a 3d program, try it out.

  The XML surrounding the numbers is probably meaningless. The numbers need to be interpreted first. Supposedly, the sets include 18 numbers per set, which seems to be the case (notice the similarity between the 1st and 4th 'vertex' entry, 18 numbers apart).

  I'll take a look at it later.

  Later: So, I realized that the numbers in this file don't make sense. I think that the CSK file basically holds the entire mesh model, possibly the bones. You'll notice that there are multiple sections with names like [ evil, good, fat, thin, strong, weak, young ] and "base". My current guess is that the 'base' section holds the actual base mesh and the other sections describe offsets of the vertices in the base mesh. However, not all vertices are referenced in each of the 'offset' sections. If you open the file in a hex editor, you'll notice that there is a dense section of information directly after the section header. This seems to be a bit-wise description of which vertices are addressed in that section.
  So, for instance, the nvidia portion of the lion CSK file shows a base mesh with 4161 vertices. The 'fat' section of the nvidia half of the CSK only references 3306 vertices. However, you'll notice that the 'dense' portion at the beginning of the fat section has exactly 4161 bits and 3306 are "1" and 855 are "0". I'm guessing that all of the vertices in the ordered list (as ordered in the base section) with a bit of '1' are referenced in that fat section and the vertex is displaced by the amount shown in the float.
  What this seems to also say is that of each 18-float number section, only the first 3 numbers are actual vertex coordinates. What are the other 15 numbers? Do they describe faces or normals in some way?

   I may not be interpreting the 4-byte numbers correctly. I assumed they were simply straight 32-bit floats, but that may not be the case.

  Again, I am not any kind of expert in this, and I really have no idea what I'm doing, hehe.. I may be entirely wrong.

-ego533
 
  Okay, first bit of success.. I have a "hard-coded" base mesh extraction for the creatures stored in .ccs files, written in python. I think there is still a long way to go before I can create a way to insert new or updated meshes. I still need to get the UV textures, I believe from the .ccs file. Somewhere, the model skeleton will need to be extracted also.  What needs looking into is how the body morph meshes are done. Finally, the hair is described in it's own way, somehow.. I have actually modified the mesh in the game and it works, but the hair 'follows' the old geometry.

  On that subject, I don't know enough about 3D mesh representations to make a decent guess at the structure of the mesh in the .csk file. I have a file attached that shows the numbers I'm looking at. They are in groups of 18. The first 3 numbers on their own line are vertex coordinates, the other related 15 numbers are on the next line. I don't know what these are. Also, the number of mesh verticies in the .csk file seems unrelated to the number of vertices found in the mesh in the .ccs file (For the wolf, there are 3020 verticies in the .ccs, 4101 in the .csk)

  A wavefront .OBJ of the wolf is attached for fun. It looks like you may need to flip Z coordinates if importing these.

 
 
   Still working on this.. The CCS file is a red-herring. It looks like this file was abandoned or otherwise not really used. Of all the information in it, only one array of bounding boxes is used in the game. Working on the CHA file currently.. to my inexperience eye,  it seems this file is an entirely proprietary format. Understanding it requires simply changing values and seeing what happens.. but I think I do have the format mapped out completely. CSK is mapped out a bit.. Still need to look at the CBN.

   Of course, even after doing this, I need to find a way to actually import/export this as a useful format to some program like Blender.. something I've also never done before. Oi..

-ego533
 
  Still no useful progress..

  Though there is a tiny chance anyone else will want to take up work on doing this, I am putting up the nearly completed description of the .CHA file. There are still two values that I still don't know how to compute and are important.. in the large array of hair sub-sections, the two-byte value that seems to describe color, transparency, and direct-light highlight are still a mystery to me.. I think they somehow describe a portion of the body texture since averaged values of 1-225 / 1-255 will paste what looks like part of the creature texture on all the hair (the eye with transparecy is visible on it). Also, in the same big array subsections, the three bytes the describe the magnitude of body-part rotation/translation effect are a mystery. I don't know how to correctly calculate these values.
  I'm hoping as I get farther in the .CSK file, these may become more obvious.

  It is pasted below as well as attached as an .RTF file with slightly better formatting. It is renamed .DOC so that it can be attached here.

Code:
   This file describes the fur for the creature.

   It contains two main sections "radeon" and "hybrid". These names are misnomers.. it isn't
really specific to the radeon, I'm pretty sure it just pertains to more powerful cards. The
hybrid section simply counts much fewer hair objects, seemingly to provide fur for less
powerful cards. It is possible that this "hybrid" section can be entirely ignored. The vast
majority of people will have powerful cards now, and alot of work can be spared by not
creating an entire second fur outline for the few people with less powerful video cards.

   The movement of fur on the body for each of the body morphs is also described in arrays
that move it around to follow the body geometry.

   I'm skipping the outline of the 'hybrid' section. Note that if you do want to edit this,
many of the numbers or arrays are not specified in this section (like the list of body
parts). I assume this section simply uses the appropriate array from the actual 'radeon'
section instead of duplicating it. (This seems odd.. most other places in these files are
not arranged for space efficiency, why do it here...)


   Though there are parts where I say things like, "If you remove values here, bad stuff
happens!" that doesn't mean you *can't* do it in all cases. You will just need to make
further adjustments to solve the problems. Like, if you remove body parts from the
material's body-part array, you'll need to re-assign or delete all the hair attached to
the part, re-organize the rotation/translation parameters in the hair section arrays,
.... 




FILE HEADER:

	String[40]
	FileHeader for the file. Usually just "LionheadCreature"

	dword[1]
	???? - Count of 4? If this is changed, the game crashes. Seems to be the same for
	all creatures.

	dword[1]
	I think this may just be some sort of version number. It seems to have no effect,
	however, it must remain "9035"



SECTION 1 - "Radeon"

	String[32]
	Name of the section

	dword[1]
	Size of this section in bytes

	dword[1]  - "MORPH_COUNT"
	number of body morphs for this creature/model

	Stringarray[MORPH_COUNT]
		String[32]
		Each string is the name of a morph, like "fat"


	dword[1] - "MORALITY_COUNT"
	number of 'morality' morphs. (Note 'gorilla' has 0. It is possible to have none. In
	this case, there are no moarilty strings or other related arrays)

	Stringarray[MORALITY_COUNT]
		String[32]
		Eachs string is the name of the moralities, usually "evil" , "good"


	dword[1]
	???????????? - Count of 3. All creatures have a count of 3 here. Changing it to 0-2
	causes hair to not be rendered. Anything above 3 creates a possibility for the game
	to crash.

	dword[1] - "BODY_PART_MAX"
	The total number of body parts in the model (hip, chest, toes, etc..)

	dword[1]
	The number of materials per major section (repeat the "material" section below this
	number of times).



	Material 

	There can be multiple materials in each section


	dword[1]
	???????????? - Count of 1. I'm *guessing* that this specifies the number of
	textures/actual materials (hair-ribbon areas) in this hair section. All creature's
	material sections always have this set to 1 (even in the 'mutiple' sections).

	UnknownChars[384]
	Material Descriptions. Currently Unknown format. 384 bytes long. This seems to
	actully have no effect. This array can be set to all 0's and nothing changes.
	I think this array is simply skipped.

	dword[1] - "BODY_PART_MATAPPLIED_COUNT"
	Number of body parts (from 0 to BODY_PART_MAX) this material is applied to.

	dword[BODY_PART_MATAPPLIED_COUNT]
	List of body parts in the range of 0 to BODY_PART_MAX that this material is placed
	on. Most arrays seem to list 'extra' parts; parts that do not actually have hair on
	them. Removing parts from this list causes the hair attached to that part to not be
	rendered, OR (since parts are referenced as an index into this array) causes hair
	to rotate/translate to the incorrect body parts).

	dword[1] - "RIBBON_SUBARRAY_SIZE"
	The size in bytes of each sub-array in the main hair-ribbon array (uually 36).

	dword[1] - "RIBBON_SUBARRAY_COUNT"
	The number of sub-arrays in the main hair-ribbon array

	dword[1]
	The size in bytes of the hair-ribbon array

		Hair-Ribbon array struct  -  "struct:hair_ribbon"
			*******
			   So, this needs explanation. To have a better idea of how this
			works, look at the pictures inside of the t_strand.dds files. Notice
			that they have an alpha channel.. look at the mask created by this
			alpha channel. Notice there are 7 lines of hair, each line has 12
			hair-types. Hairs can be up to 7 'sections' in length.
			   Evidently, this works by describing hair
			in varying sizes, directions, colors, luminosity. The hairs are
			described by sections. The longer hairs have up to 7 sections from
			the base to the tip. Each hair section is described by TWO entries,
			the values and meanings of some of the variables being averaged.
			The longest hairs are described by 14 entries (7 sections). This
			means that, for instance, for the second 'section' of a long hair
			ribbon, you can make the darkness variable either the same for both
			entries describing this section, making it very dark or very light,
			or you can make the two darkness numbers different and the color
			will be the average of these two ( grey instead of light or dark ). 

			float[3]
			Normal describing the direction the hair-ribbon points. This is
			in 'model-world coordinate' space, not relative to the
			perpindicular placement of on the mesh. A normal of 0,0,0 seems
			to point 'back' and 'down' at a 45 degree angle.

			byte[4]
				byte[1 , 4]
				Note this is byte 1 and 4, not 1 through 4. These seem to
				describe how much of the ambient light is taken in by the
				hair section. 0 is little, higher numbers are more. This
				does NOT mean 'darker' or 'lighter.' If the ambient light
				is dark, the highlight is dark, and so forth. This is not
				the 'directed' light, just the ambient light. For instance,
				ribbons with a high number here will pick up a strong
				orange glow duing sunsets *no matter* the direction the
				ribbon is facing or whether it is in shadow. The directed
				lights, like from sun or torches don't seem to affect the
				highlight at all. Oddly, only byte 1 seems to have any
				effect. If different from byte 1, there is no effect. In
				all cases I see, these two bytes always match. retail
				numbers used are 0, 64, 112, 147, 174, 194, 210. The
				default is to make each hair section one tick higher
				( 0 -> 64 -> 112 -> 147 ...). The base section is 0.

				byte[2 , 3]
				Information about the color, luminosity, transparency and
				highlight information. I'm not quite sure how this all ends
				up being done in 16 bits.. I do not understand how these
				numbers are calculated... Unfortunately this is important..
				Not knowing how to calculate these means that all hair
				color and visibility cannot be controlled.


			byte[5]
			So, this works a little oddly. I'm not sure quite why or how these
			numbers are caculated. This is basically a set of three numbers
			which specify which body parts will affect rotation/translation of
			the hair ribbon as the creature moves, inluding the interpolated
			stretching and contracting of the skin. These three numbers are
			seperated with two 0's. The magnitude of affect of each specified
			body part is specified in the next set of numbers. In general the
			body part the ribbon is actually attached to is specified in the 
			first number. Finally, the way the body parts are referenced is a
			bit odd. All numbers are multiples of 3. They reference body parts
			not directly as the body part number, but from the array of
			body-parts specified in this material section above. 0 is te first
			body part in this array, 3 is the second, 6 is the third, 15 is
			the fifth, .... So (using numbers from the wolf), the material for
			the head-hair has the body part array beginning with
			[ 2 3 4 5 6 7 8 9 ...], so if the hair ribbon is on the back of
			the neck, the array would probably be [ 3 0 6 0 9 ] which
			references the body part 3, 4, and 5 (3=neck 4=back-neck-scruff
			5=head). The movement and skin movement of these three bdy parts
			will affect the movement of the ribbon. Finally, it is possible
			that these are words (not dwords) instead of bytes. That would
			explaing the 0's... the first 0 in the following 3-bytes would
			then be part of the third number.


			byte[3]
			??????? - Padding? These seem to always be 0, but do in fact affect
			movement of the ribbon. I think this can safely be igonred and
			always set to 0.


			byte[3]
			These three bytes should add up to 255. These numbers define the
			magnitude of the effect movement and skin movement of the body
			parts specified in the 5-byte section have on this hair-ribbon's
			movement. These numbers are in reverse order. The first number in
			the 5-byte is affected by the third number in this set, the
			second by the second, and the third byte in the 5-byte section by
			the first byte in this section. If these numbers don't add up to a
			total od 255, the ribbon will screw up. The numbers can be set to
			0, like [ 0 0 255 ] in which case te ribbon will follow only the
			first bodyart specified in the 5-byte section (usually, this will
			screw up following skin movement). I'm not at all sure how these
			numbers are calculated. I'm hoping it will become obvious after
			decoding the other files. It may have something to do with
			percentage overlap of body-part bounding boxes which I do not look
			forward to calculating if is the case...


			byte[1]
			??????? - Padding? Alwasy 0, seems to have no effect.


			byte[8]
			Only the first 7 bytes are used (the retail hair-ribbon texture
			files used only have 7 rows, so the 8th bit is unused). There are
			only two values used, 0 and 255 (or -1). These indicate to which hair
			'section' the line applies. In almost all cases, each section is
			specified by two lines apiece. The order of section specification is
			not obvious: [ 3 2 1 4 7 6 5 NA ]. So, for example, a 2-section hair
			ribbon will usually have these as [0 0 255 0 0 0 0 0],
			[0 0 255 0 0 0 0 0], [0 255 0 0 0 0 0 0], [0 255 0 0 0 0 0 0]. It
			actually works to specify this out of order, but causes odd shapes in
			the hair.



	struct:hair_ribbon[RIBBON_SUBARRAY_COUNT]
	Big, long, array of hair ribbon sections...

				

	dword[1] - "RIBBON_COUNT"
	The number of 'hair ribbons' in this material section. This is the number of ribbons,
	not the number of hair sections (which is instead RIBBON_SUBARRAY_COUNT)

	dword[1] - "MORPH_COUNT_PLUS_BASE"
	This is the number of body morphs this model has plus 1 (for the 'base' morph).


	floatarray[RIBBON_COUNT]
		float[6]
		Two points specified as (x,y,z) coordinates.
	These are arrays of vertices that determine where the hair is placed. The coodinates use
	'creature-world' coordinates, the same used to construct the model. This array describes
	the placement of aech hair ribbon. Two points are specified per 'line'
	(RIBBON_COUNT * (3)*(2) floats). The line formed between these two points specifies
	where a hair ribbon will be placed.
	

	floatarray[RIBBON_COUNT][MORPH_COUNT]
		float[6]
		Two points specified as (x,y,z) coordinates.
	These arrays are similar to the array above. Each of the arrays describes the
	*displacment* of each vertex for the corresponding morph. The morphs are in the order as
	specified at the begining of the file where the morphs are named. There are two
	point-displacment specifications per line (6 floats) which 'move' both side of the line
	describing a hair ribbon.


	dword[1] - "RIBBON_COUNT"
	Number of hair ribbons. Identical to the one above.

	dword[RIBBON_COUNT]
	This is an array describing the number of 'ribbon sections' per hair ribbon. As
	described above, hair can have up to 7 sections for length (usually comprised of 2 lines
	apiece in the struct:hair_ribbon). In an ordered list corresponding to the order in the
	struct:hair_ribbon, each dword gives the number of sections for that ribbon

	dword[RIBBON_COUNT]
	This array appears to describe the texture quality placed on the hair ribbon. Each
	ribbon is assigned a quality number 0, 1, or 2. 0 = low, 1 = med, 2 = high. Lower
	qualities look a little more fluffy, but do become obviously blurry. Usually, the short
	hairs are made low quality (I assume to avoid high-quality rendering when not really
	necessary)

	dword[1] - "HAIR_UV_ROWS"
	This is more of a guess than other values. I think this is either the number of maximum
	hair sections or the number of rows of hair types as found in the t_strand.dds files
	(These two numbers should generally be the same anyway). See the array notes for details. 

	dword[1] - "HAIR_UV_TYPES_PER_ROW"
	This is more of a guess than other values. In each row of similar-length hair in the
	t_strand.dds files, there are groups of hair that are very similar in appearance. I think
	this number describes how many groups of similar hair on each row there are.

	dword[1] - "HAIR_UV_PIECES_PER_GROUP"
	This is more of a guess than other values. I think this desribes the number of individual
	hair tesxtures per 'group' as defined in "HAIR_UV_TYPES_PER_ROW".

	********
	Rather than try to make a 'structure' for this array, I'm just going to explain how I
	think it works...

	lol, math
	Let f(x) = Summation of 1 to x for:
	([2*x] + 2) * (HAIR_UV_TYPES_PER_ROW) * (HAIR_UV_PIECES_PER_GROUP)
	then:
	********
	------> float[ f(HAIR_UV_ROWS) ] <---------
		This array describes UV points that define individual hair pieces in the hair
	texture files (t_strand.dds). UV points are 'percentages' of the max 2D X,Y coordinates
	dimensions of the picture file, so values range from 0 to 1 in both X,Y. The structure of
	this is not completely straightforward. There are "HAIR_UV_ROWS" rows in the texture file.
	I suspect the behavior of this row count is to add one hair 'section' per row in the .dds
	file. For instance, the first row has one section desribed by 4 points (a box). The second
	row is described by 6 points (two squares/boxes). The third row is described by 8 points
	(three boxes). The 'boxes' define the hair sections that are used and referenced in the
	hair length in the big "struct:hair_ribbon" above. On each row, there are
	"HAIR_UV_TYPES_PER_ROW" groups of "HAIR_UV_PIECES_PER_GROUP" hairs. For example, the
	retail texture files have 7 rows, 3 groups, 4 pieces-per-group. That means 7*3*4=84 hairs.
	Rows 1-7 have 1-7 sections-per-hair respectively, described by (2n+2) points for a total of
	840 points, described by two floating point numbers apiece = 1680 numbers, each floating
	point is 4 bytes, for a total array size of 6720 bytes. incidently, the boxes are
	described by a 'Z' pattern.. Top-left, Top-right, Bottom-left, Bottom-right.



	dword[1] - "MORALITY_COUNT_PLUS_BASE"
	I belive that this count will be "MORALITY_COUNT" + 1. This specifies the number of
	different ribbon-length numbers to specify for each ribbon referenced in the following
	array, one for each 'morality' plus a base number
	

	floatarray[RIBBON_COUNT][MORALITY_COUNT_PLUS_BASE]
		float[1]
	This array seems to specify the length of each hair ribbon. The first array of
	"RIBBON_COUNT" floats is the length of each hair ribbon in the base model, the subsequent
	arrays define the hair length for each of the morailty sections (the order is determined by
	the order of morailty section specificaiton in this major section's header). The maximum
	length varies by the number of sections a hair ribbon has. beginning at "0.14285715" for a
	two-section ribbon, each seaction added adds ~"0.14285715" more to the maximum length (some
	rounding error, to be safe, add only "0.14285714") So, for instance a 7-section ribbon has
	a maximum length of "0.85714289". I have no idea why these are the numbers. They don't
	correspond to anything I see. I presumed they would be some sort of distance in the
	hair-texture (t_strand.dds) file, but they aren't. They seem entirely arbitrary.
	actual max's to 8-digit precision accounting for precision errors (starting with 2-section
	ribbons) are [ 0.14285715 0.28571431 0.42857144 0.57142862 0.71428576 0.85714289 ]


	dword[1] - "RIBBON_COUNT"
	The number of hair ribbons. Identical to the ones above.

	floatarray[RIBBON_COUNT]
		float[3]
	This array consists of one 3D point per-hair ribbon. The point is used to determine the
	'Z' rendering of the ribbon.. In other words, which ribbons appear in 'front' of others.
	Ribbons calculated to be in front will be shown to the camera, those behind will be hindden
	behind those in front. It seems that the points are generally simply set to the first vertex
	in the *base model* array of two-verticies-per-ribbon that create the hair ribbon positions
	(the one before the other MORPH_COUNT arrays of dual-points).

	dword[RIBBON_COUNT]
	This array attaches each hair ribbon to a particular body part. In an ordered list accroding
	to the "struct:hair_ribbon" array, the dword specifies a number found in the array of body
	parts this material is attached to. The specified numbers is the 'actual' body part number,
	not an index into the array of body parts, but the body part will be specified in the array.

	dword[RIBBON_COUNT]
	This array contains numbers that count from 0 to RIBBON_SUBARRAY_COUNT. The count follows
	the ordered list of RIBBON_COUNT hair ribbons, each number for that indexed ribbon
	corresponds to the index point at which the hair sections begin to be referenced in the 
	"struct:hair_ribbon" array. (Keep in mind that each hair section includes two
	lines-per-section). For instance, if there are 3 hair ribbons 1 = 7 sections, 2 = 4 sections,
	4 = 6 sections, the array would be [ 0, 14, 22 ]. The first ribbon has it's sections
	beginning to be referenced at index point 0, the second at index point 14 (since the first
	ribbon takes 14 lines to specify the 7 sections), the third at index 22.

	dword[RIBBON_COUNT]
	Related to the array directly above. This array specifies the number of hair-section lines
	per hair ribbon, following the ordered list of hair ribbons in the "struct:hair_ribbon"
	array. From the example above (3 hair ribbons 1 = 7 sections, 2 = 4 sections,
	4 = 6 sections), the array would be [ 14, 8, 12 ].

	dword[1] - "RIBBON_SUBARRAY_COUNT"
	Identical to the one further up. I don't know why this is specified here.

	************ Repeat material section *******************
 
 
   Just have an urge to comment/complain, sorry: There is alot of unused information in these files. It looks like there were several false-starts and abandoned ideas in how these models were constructed. It's making it confusing to unravel the files, also, since there are numbers of arrays and useful-seeming information that ends up being irrelevant..

   The pictures of extracted models I had in the other thread I believe are correct, but I've since found out that the actual place I extracted face information from is another abandoned array.. that makes the third irrelevant mesh-face array I've found, oi. There is one array I don't yet understand in the .CSK file that could be the actual face array, but it seems to make use of body-part definitions I haven't found yet.

   In short, it will be a little longer before I find the correct spot to update data with, hehe. I'm hoping it is in the CBN file since that's the only file left to parse.. If not, I'm missing something fundamental about the way these models are stored.

-ego533
 
 
I have noticed a trend when following people's efforts to mod games.

As a coder by profession, I know it's normally best practice to keep your code tidy in the likelihood it may be used again in the future.  However if I were a bettin' man, I'd say that the game developers don't seem to follow this practice for a few reasons:
  • The game's alot less likely to have it's code re-used once released, so it's not as important to keep it clean.  Generally, you also need to be careful cleaning up redundant code as you might break something useful.  Messy code that works is better than tidy code that doesn't.  The only possible exceptions to not reusing game code would of course be a sequel that's not developed from the ground up and modding (which isn't really a concern when a game is being developed).
  • Time constraints.  Cleaning up redundant code eats up valuable time.  And the smaller developers (like Lionhead was at the time) feel this pressure even more so.

In short, this means you gotta trudge though more than your fair share of dead ends in figuring out the code :suspect .
 
  Still working, hehe.. of course, I realized that everything is more complicated than I had thought. I'm at that point where I now know enough to realize how much I don't know and how much work there is to do to get this right. I planned to generalize the abilities to import/export models, but I may give that up and make it more directed like the model import/extract for BW1..

  As a bit of show for progress, here is the so-far completed work on a simple creature model exporter. This exports the creature models to an OBJ file with imporvised polygroups. Note that the BW2 models have many duplicate vertices over UV texture seams ( I assume to help shading/lighting.. you need multiple tangent values at UV seams to get correct shading). These duplicates can cause problems in modelling programs if not welded, but if you weld, you also then screw up the later import back into the game since the game will expect the duplicates (unless you re-duplicate..). Also, many 3D programs use different coordinate systems for both XYZ and UV (3D space and texture mapping). Most need these models to have their Z inverted. Some require you to flip the UV to get the right direction (Z-brush..). Also, the UVs as used in the game are actually mirrored and squashed, so the UV map for the model reflects this. You can choose to correct the exported UVs so that they work correctly on the exported model using the pictures extracted from the dds game files.
  This is a console python script.. You will need to have Python installed. I plan to make a 'compiled python' version as an exe at some point.
   Have a copy of all the creature files ( CCS  CBN  CHA  CSK) in the same folder. Run the script with an argument of any one of the files. For example: [ > scriptname.py bwolf.cbn ]

Code:
#
# Black and White creature model exported V0.1
#
# You must have Python installed for this to work. Written around Python 2.7.
#
# This is NOT a 'well' written thing and probably will not be.. I'm not a programmer..
# This script takes a Black and White 2 creature model and exports an OBJ with
# polygroups. All four of the creature model files must be in the same folder!
# NOT TRUE YET (V0.1): You can simply drag any of the four files onto this script file in the OS and an
# OBJ will be created in the same folder.
#
# I tried creating this complicated 'weighting' method for extracting polygroups
# from the CSK. It kinda works. I have a better idea of how to do it, that I
# think will extract better polygroups, but it is also complicated. I'm not
# sure it is worth getting polygroups, anyway. If people find them useful, let
# me know and i'll see if te other method does a better job (kayssplace.com).
#
# a " [] " is an array (pointer), a " () " in pythin is a 'Tuple'.
#
# For reference, Data structures:
# Mats, for each material, there will be the following
# Mats = [ array-count, [(partarray), [num-parts,index,count] , ..repeat..]
#                        (partarray)  [num-parts,index,count]
#                        (partarray)] [num-parts,index,count]
#
# VertexTuple, UVTuple, faceTuple simply contains each sub-array from their arrays in the
# CSK file unpacked into their correct data types (floats, words, dwords)
#


import struct
import sys
import os

######## Arguments #######################
if len(sys.argv) < 2:
	print "Missing Argument:  A CCS file"
	print
	print "Usage:  CCS_AllValues.py  bcreature.CCS"
	raise SystemExit

filename = os.path.basename(sys.argv[1])
filedir = os.path.dirname(sys.argv[1])
if filedir != "":
	filedir += "/"
filenamenoext, filenameext = os.path.splitext(filename)
pathtomodelnoext = filedir+filenamenoext
print filenameext
if not os.path.isfile(filedir+filenamenoext+filenameext):
	print
	print "ERROR:File in argument does not exist"
	print
	raise SystemExit
if not os.path.isfile(pathtomodelnoext+".ccs"):
	print
	print "ERROR:CCS file does not exist"
	print
	raise SystemExit
if not os.path.isfile(pathtomodelnoext+".cha"):
	print
	print "ERROR:CHA file does not exist"
	print
	raise SystemExit
if not os.path.isfile(pathtomodelnoext+".csk"):
	print
	print "ERROR:CSK file does not exist"
	print
	raise SystemExit
if not os.path.isfile(pathtomodelnoext+".cbn"):
	print
	print "ERROR:CBN file does not exist"
	print
	raise SystemExit

try:
	orfile = open(pathtomodelnoext+".csk", 'rb')
except:
	print
	print "ERROR:Cannot open CSK file for reading"
	print "Why? File seems to exist. Persmission problems?"
	print
	raise SystemExit


try:
	owfile = open(pathtomodelnoext+".obj", 'w')
except:
	print
	print "ERROR:Cannot open file for writing."
	print "Disk Full? Directory read-only?"
	print
	raise SystemExit
##########################################



# i j k will be used as a loop index
i = 0
j = 0
k = 0
m = 0
h = 0
userInput =''
#Negate Vertex Z
NVZ = 1
#Negate UV's (misnomer.. not really negating, just re-arranging)
NUV = 0
#Negate Normal Z
NNZ = 1
#Should the model be welded?
weld = 0
#Should the morphs be extracted?
extractmorphs = 0


###Ask if this is a retail model. If so, run the complicated matching with the
###CCS file to get good group mappings
print
print
print "CTRL-C at any point to break this script."
print
print
print
print "Should Vertex Z be inverted?"
print "  1) Y"
print "  2) N"
print
while userInput != "1" and userInput != "2":
	userInput = raw_input('--> ')
	if userInput != "1" and userInput != "2":
		print "Input is not correct"
if userInput == "1":
	NVZ = -1

userInput = ''
print
print
print "Should Normal vector Z be inverted?"
print "  1) Y"
print "  2) N"
print
while userInput != "1" and userInput != "2":
	userInput = raw_input('--> ')
	if userInput != "1" and userInput != "2":
		print "Input is not correct"
if userInput == "1":
	NNZ = -1

userInput = ''
print
print
print "Should UV's be arranged to match .dds texture files?"
print "If this is not done, the texture files will not inherently"
print "match exported model. See readme.txt for explanation."
print "  1) Y"
print "  2) N"
print
while userInput != "1" and userInput != "2":
	userInput = raw_input('--> ')
	if userInput != "1" and userInput != "2":
		print "Input is not correct"
if userInput == "1":
	NUV = 1

userInput = ''
print
print
print "THIS IS NOT YET IMPLEMENTED. ANSWER IGNORED"
print "Should the model be welded?"
print "This will also affect the morphs, if extracted."
print "This will remove duplicate vertices. Shading may"
print "need these for tangent infromation along UV seams"
print "but not welding will cause problems for model use"
print "outside of the game.. See the Readme.txt"
print "  1) Y"
print "  2) N"
print
while userInput != "1" and userInput != "2":
	userInput = raw_input('--> ')
	if userInput != "1" and userInput != "2":
		print "Input is not correct"
if userInput == "1":
	weld = 1


userInput = ''
print
print
print "THIS IS NOT YET IMPLEMENTED. ANSWER IGNORED"
print "Should the morphs be extracted?"
print "The output filename will be"
print "modelname-morphname.obj."
print "Note that if you plan to re-import these"
print "morphs, these specific filenames will be"
print "looked for."
print "  1) Y"
print "  2) N"
print
while userInput != "1" and userInput != "2":
	userInput = raw_input('--> ')
	if userInput != "1" and userInput != "2":
		print "Input is not correct"
if userInput == "1":
	extractmorphs = 1




###Get the maximum body part count
orfile.seek(52)
valueBuffer = orfile.read(4)
BODY_PART_MAX = struct.unpack_from('<i', valueBuffer)[0]


###If the first section is not 'radeon', Skip it. In that case,
###Gets the size of the first section, then jumps ahead that
###number of bytes.
valueBuffer = orfile.read(32)
if valueBuffer[1] != "r" and valueBuffer[1] != "R":
	valueBuffer = orfile.read(4)
	scratchvalue = struct.unpack_from('<i', valueBuffer)[0]
	###Skip the section
	orfile.seek(scratchvalue, 1)
	###Skip the second section's header
	orfile.seek(36, 1)
else:
	###If the section IS 'radeon', skip the section-size value
	orfile.seek(4, 1)






Mats = []
arraysRead = 0
###The Mat array data structure is documented at the top of this file.
###
valueBuffer = orfile.read(4)
NUM_MATS = struct.unpack_from('<i', valueBuffer)[0]

i = 0
while i != NUM_MATS:
	orfile.seek(516 ,1)
	valueBuffer = orfile.read(4)
	NUM_PART_ARRAYS = struct.unpack_from('<i', valueBuffer)[0]
	Mats.append(int(NUM_PART_ARRAYS))
	Mats.append([])
	Mats.append([])
	j = 0
	while j != NUM_PART_ARRAYS:
		valueBuffer = orfile.read(4)
		PART_ARRAY_COUNT = struct.unpack_from('<i', valueBuffer)[0]
		Mats[(i*3)+2].append(PART_ARRAY_COUNT)
		### Read the array and the two count numbers
		valueBuffer = orfile.read(PART_ARRAY_COUNT * 4)
		Mats[(i*3)+1].append(struct.unpack_from('<'+str(PART_ARRAY_COUNT)+'i', valueBuffer))
		valueBuffer = orfile.read(8)
		Mats[(i*3)+2].append(struct.unpack_from('<i', valueBuffer)[0])
		Mats[(i*3)+2].append(struct.unpack_from('<2i', valueBuffer)[1])
		arraysRead += 1
		j += 1
	i += 1






###We are now at the beginning of the Face array.
###For convenience, this should be done later. Push
###The file address
valueBuffer = orfile.read(4)
FACE_ARRAY_ENTRY_COUNT = struct.unpack_from('<i', valueBuffer)[0]
faceArrayLoc = orfile.tell()
###The entries in this array are words (2 bytes)
orfile.seek(FACE_ARRAY_ENTRY_COUNT * 2 ,1)





###Read in the UV/weight array. It is neede to
###split the vertices into groups
valueBuffer = orfile.read(8)
VERT_META_ARRAY_SUBARRAY_SIZE = struct.unpack_from('<i', valueBuffer)[0]
if VERT_META_ARRAY_SUBARRAY_SIZE != 28:
	print
	print "ERROR:UV array in unknown format"
	print
	orfile.close()
	owfile.close()
	raise SystemExit
VERT_META_ARRAY_SIZE = struct.unpack_from('<2i', valueBuffer)[1]
UVArrayLoc = orfile.tell()
orfile.seek(VERT_META_ARRAY_SIZE ,1)







vertex = "\n"
vertexNormals = "\n"
VertexTuple = []
###Begin extracting the vertices
orfile.seek(32 ,1)
valueBuffer = orfile.read(8)
BASE_VERTEX_ARRAY_SUB_ARRAY_SIZE = struct.unpack_from('<i', valueBuffer)[0]
if BASE_VERTEX_ARRAY_SUB_ARRAY_SIZE != 72:
	print
	print "ERROR:Vertex array in unknown format"
	print
	orfile.close()
	owfile.close()
	raise SystemExit
BASE_VERTEX_ARRAY_SIZE = struct.unpack_from('<2i', valueBuffer)[1]
MODEL_VERTEX_COUNT = BASE_VERTEX_ARRAY_SIZE/BASE_VERTEX_ARRAY_SUB_ARRAY_SIZE
vertexBuffer = orfile.read(BASE_VERTEX_ARRAY_SIZE)
j = BASE_VERTEX_ARRAY_SIZE
i = 0
while i != j:
	VertexTuple.append(struct.unpack_from('<18f', vertexBuffer[i:i+BASE_VERTEX_ARRAY_SUB_ARRAY_SIZE:1]))
	i += BASE_VERTEX_ARRAY_SUB_ARRAY_SIZE
i = 0
###Since the vertex count is absolute, but i is an index that
###begins with 0, no need to add one to the loop end
while i != MODEL_VERTEX_COUNT:
	###NVZ is the Negate Vertex Z value. This is set by the user question at the beinning. If no, this is "1"
	###so no inversion, otherwise -1 which inverts. Same for the normal.
	vertex += "v " + str(format(VertexTuple[i][0], ' .8f')) + " " + str(format(VertexTuple[i][1], ' .8f')) + " " + str(format((VertexTuple[i][2]*NVZ), ' .8f')) + "\n"
	vertexNormals += "vn " + str(format(VertexTuple[i][3], ' .8f')) + " " + str(format(VertexTuple[i][4], ' .8f')) + " " + str(format((VertexTuple[i][5]*NNZ), ' .8f')) + "\n"
	i += 1
i = 0
owfile.write(vertex)
owfile.write(vertexNormals)

del vertex
del vertexNormals
del VertexTuple






vertexUV = "\n"
UVTuple = []
###Go back and get the texture coordinate array
orfile.seek(UVArrayLoc)
vertexBuffer = orfile.read(VERT_META_ARRAY_SIZE)
j = VERT_META_ARRAY_SIZE
i = 0
while i != j:
	UVTuple.append(struct.unpack_from('<2f4h3f', vertexBuffer[i:i+VERT_META_ARRAY_SUBARRAY_SIZE:1]))
	i += VERT_META_ARRAY_SUBARRAY_SIZE
i = 0
###Since the vertex count is absolute, but i is an index that
###begins with 0, no need to add one to the loop end
while i != MODEL_VERTEX_COUNT:
	###If the user opted to corrected the UV's, do it.
	if NUV:
		if UVTuple[i][0] > 0.5:
			vertexUV += "vt " + str((1 - UVTuple[i][0]) * 2) + " " + str(1 - UVTuple[i][1]) + "\n"
		else:
			vertexUV += "vt " + str(UVTuple[i][0] * 2) + " " + str(1 - UVTuple[i][1]) + "\n"
	else:
		vertexUV += "vt " + str(UVTuple[i][0]) + " " + str(UVTuple[i][1]) + "\n"
	i += 1
owfile.write(vertexUV)


del vertexUV
###Don't delete the UVTuple since it is still needed.. I think most people have plenty
###of memory and theses arrays aren't THAT big..





#"faceTuple" declared below to hold the face array
partFaceArray = []
weightHelp = []
numArrays = 0
endArrayEntryNum = 0
highestWeightValue = -1
highestEntryNum = -1
fIndex = 0
fCount = 0
###Get the faces. this is a little complicted. I decided (for now)
###to only assign one body part to each face. If I am able to extract
###bones, the way this works may change since each vertex will be
###weighted to multiple body parts
###
###Create "BODY_PART_MAX" part arrays. Faces will be added to these
###Note that these will be indexed from 0, so the actual index numbers
###will be BODY_PART_MAX - 1.
###Also create weighthelp array. This is used to collect vertex
###weights to help detemine face group
orfile.seek(faceArrayLoc)
###Each entry is a word, 2-bytes
valueBuffer = orfile.read(FACE_ARRAY_ENTRY_COUNT * 2)
faceTuple = struct.unpack_from('<' + str(FACE_ARRAY_ENTRY_COUNT) + 'h', valueBuffer)
i = 0
while i != BODY_PART_MAX:
	partFaceArray.append("g BodyPart_" + str(i) + "\n")
	weightHelp.append(int(0))
	i += 1
i = 0
while i != NUM_MATS:
	###numArrays will be the number of part arrays in this Material
	numArrays = Mats[i*3]
	j = 0
	while j != numArrays:
		###k will be the beginning index of this part-array's listings in the face array
		k = Mats[(i*3)+2][(j*3)+1]
		###endArrayEntryNum will be the last index number in the face arary for this
		###part-array
		endArrayEntryNum = Mats[(i*3)+2][(j*3)+2] + k
		###Faces will be triplets, (index-1, index, index+1). For index-1 and index+1 to
		###exist correctly, the reading must start at k+1 and end at endArrayEntryNum-1
		k = k + 1
		endArrayEntryNum = endArrayEntryNum - 1
		h = 0
		while k != endArrayEntryNum:
			m = 0
			###reset the weight array buckets
			while m != BODY_PART_MAX:
				weightHelp[m] = 0
				m += 1
			###reset highest index weight tracker
			highestWeightValue = -1
			highestEntryNum = -1
			###As long as any of the numbers in the triplet are not the same...
			if faceTuple[k] != faceTuple[k-1] and faceTuple[k] != faceTuple[k+1] and faceTuple[k-1] != faceTuple[k+1]:
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k]][2])/3]] += UVTuple[faceTuple[k]][6]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k]][3])/3]] += UVTuple[faceTuple[k]][7]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k]][4])/3]] += UVTuple[faceTuple[k]][8]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k-1]][2])/3]] += UVTuple[faceTuple[k-1]][6]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k-1]][3])/3]] += UVTuple[faceTuple[k-1]][7]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k-1]][4])/3]] += UVTuple[faceTuple[k-1]][8]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k+1]][2])/3]] += UVTuple[faceTuple[k+1]][6]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k+1]][3])/3]] += UVTuple[faceTuple[k+1]][7]
				weightHelp[Mats[(i*3)+1][j][(UVTuple[faceTuple[k+1]][4])/3]] += UVTuple[faceTuple[k+1]][8]
				m = 0
				while m != BODY_PART_MAX:
					if weightHelp[m] > highestWeightValue:
						highestWeightValue = weightHelp[m]
						highestEntryNum = m
					m += 1
				if h == 0:
					partFaceArray[highestEntryNum] += "f " + str(faceTuple[k+1]+1)+"/"+str(faceTuple[k+1]+1)+"/"+str(faceTuple[k+1]+1) + " " + str(faceTuple[k-1]+1)+"/"+str(faceTuple[k-1]+1)+"/"+str(faceTuple[k-1]+1) + " " + str(faceTuple[k]+1)+"/"+str(faceTuple[k]+1)+"/"+str(faceTuple[k]+1) + "\n"
				else:
					partFaceArray[highestEntryNum] += "f " + str(faceTuple[k]+1)+"/"+str(faceTuple[k]+1)+"/"+str(faceTuple[k]+1) + " " + str(faceTuple[k-1]+1)+"/"+str(faceTuple[k-1]+1)+"/"+str(faceTuple[k-1]+1) + " " + str(faceTuple[k+1]+1)+"/"+str(faceTuple[k+1]+1)+"/"+str(faceTuple[k+1]+1) + "\n"
			h = (h + 1) % 2
			k += 1
		j += 1
	i += 1
###Oi... So, now all of the groups have been collected.. write them out
i = 0
while i != BODY_PART_MAX:
	owfile.write(partFaceArray[i])
	i += 1


del UVTuple
del partFaceArray
del weightHelp
del Mats
del faceTuple


orfile.close()
owfile.close()

-ego533
 
Wow, all this stuff about extracting the models and modifying them is really impressive. The closest I've come to working with models is my accidental discovery of FEATURE_INFO_POINTER_HAND (aka. the AWESOME, flying pink heart!!! Totally manly and awesome.) but that's not even related to the stuff you're doing. Keep up the good work!

Edit: $hit I was drunk when I posted that...
 
  Oi, can't belive it's been almost 7 months.. odd events in the last couple years.

  I actually got quite a bit farther than my last post shows. There probably isn't much interest in this but I've taken it on as part of a a personal hobby, basically.. There are no promises that much more will be done.

  I have a generalized extraction script completed (still 'dirty'), and nearly have an import script complete that equals the functionality of the BandW 1 milkshape plugin.. have to keep the same bones, the same number of vertexes, and the same vertex order. The one last problem in the import script is nailing down the method of packing the face array. It looks like there was an attempt to pack the model face array using a sliding window method, with many unused triplets created during the packing, and I have to  figure a mathod to re-pack the face array in the same way. It' straight-forward, I just need to think about it for awhile..

  To get beyond the restrictive import requirements is a huge leap in complexity.. Bone weights will have to be figured out so that new mesh vertexes can be added to the skeleton correctly, but the real difficult part is the animations. I suspect that animations include not only bone translation but mesh deformations.. If that's the case, then it may be that modifying the model meshes beyond these restrictions will ruin all of the existing animations for the model.

-ego533
 
This is some great work you're doing, I think it's gone more in depth than anyone else's forays in to BW2 before so well done  :) It sounds so much more complicated than BW1 but then i guess it's going to be  :D

What I think might be really appreciated (and probably easier by quite a margin) would be to make an importer and exporter for BWM models, i.e. buildings and the sort. I'm not sure what you can do with that but I wonder if it's possible.

Either way keep up the good work, I think we're all pretty baffled already  :D
 
  I have some information on the BWM files. I'll see if it is possible, subject to the caveat of motivation..

  I now have a working import and export script for the creature mesh. I still have no bone or animation information. In addition, similar to the old Milkshape plugin for black and white 1, the mesh topology cannot be changed, only the geometry.. no vertices can be added or removed. Finally, the vertex order must also be maintained.

  So, I solved the wolf face problem and I think I'll be able to upload a fix today. It is sort of silly.. whoever made the model just made a dumb mistake in the way they entered the morphs. There is a 'base' model file to which each morph like 'thin' or 'good' apply. For the wolf, the 'base' model was accidentally entered as the 'young' morph instead of the actual base model. I was able to take a model that is approximately halfway between the 'weak' and 'strong' morphs that seems to work well for the actual base model. Doing this has a pleasant side-effect of seeming to also fix the face problem. So, in short, the adult wolf that you see in the game isn't really the adult wolf, it's the young wolf just bigger. Waxing dramatic: we shall see the adult wolf for the first time! lurlz! Or, at least an approximation..


Fix uploaded: [ http://www.bwfiles.com/files/?display=1372 ]

-ego533

EDIT:  So close, but not quite.. there's a specular lighting problem with the morphs that my script produces.. I will work to fix this first. I have to say, the Good wolf looks quite a bit less "derpy".

Edit2: Problem solved. I generated the new morph files and tested them out. Seems to work. A mod file has been uploaded to the kayssplace server for BW2 Mods.

 
FYI - Our uploads system is a bit messy.

You need to find the right mechanism that automatically uploads your mod to the correct download category.  What I fear is that you submitted the file to an older upload service where kays has to review it and then put in the downloads section himself.

As he's not around so much, that may never happen :( .

If you're mod is showing up in the downloads section, I would suggest putting a link in this topic just to be helpful (I can't seem to find it myself).
 
   I was wondering if something like that would be the case.. I searched out the correct method and uploaded the file again.

-ego533
 
 
Brilliant.  I see it now. 

Glad you were able to find the right method.
 
   I don't know why I post when I really have nothing new to share.. I think detailing progress a bit aids in keeping my motivation for this hobby up. Though, it looks more and more like it's a project who's fruits will be enjoyed by 0 to 3 people, hehe... I see why code-breaking is enjoyable to some. This seems like the same type of challenge, but with a contemporary-nerd twist to it. Also has taught me the basics of how 3D computer programming works.

   I made some further progress in decoding the creature files. Unfortunately, it is detail work. As I generalized the scripts more, it became apparent that several assumptions made about structures were wrong. I've gone back and re-tested and fixed those sections, I hope. It looks like the CCS file may not be as abandoned as I thought, either. In working with the edge connection sections in the CSK (which I had thought were abandoned), there is some pattern of relation to portions of the CCS file. Most of the CCS file except the bounding box array doesn't seem to actually affect the game (the file can actually be made 0-sized except for the structure and that one array), but the data in it may NOT be old or unrelated to the actual game model in the CSK. I realized how the model in the CCS may be able to be related to the model in the CSK file which would be great since the CCS file has neatly laid out groups defining vertex-bone attachment. This also means that a proper import script should also update the CCS file, though, which is an entire new project.

   I started looking at the BWM files. Actually extracting the model looks easy, but it's the other data in the file that may be difficult to work with. So far, it looks like updating the model of static things like buildings with no animations may be possible, but I don't know how much may really be done to them.

-ego533

 
 
Whist right now I am to tired to actually read through everything, the basics of what I saw seems to be a breakthrough of the likes we haven't had for quite a while! The last being of Keatosn or what ever her name was, the german modder? - I'd like to see where this ends up! :)
 
   I decoded the structure (mostly) of the LUG audio files. I've come up with a few ways to possibly modify the creature hair files, but likely not in a way that is acceptable to release publicly.. it will require quite a bit of manual intervention and knowledge. I'm thinking to try and create an automatic "hair-fixer" thing that would rearrange hair on the creature to match moderate mesh alterations.

   I've been looking at the BWM files. They aren't going to be easy to decode. The actual model is easy. The difficult part is the textures, shaders, and the way the buildings change with alignment. The inclusion of these creates a format that changes within each BWM file. I don't understand yet how the file 'knows' what is what among these or how they are read.

-ego533

 
 
Back
Top