Olio-ohjelmointi Opasnetissä

Tämä sivu on tietokide alatyypiltään metodi. Sivutunniste: Op_fi2934
Moderaattori:Ei ole (katso kaikki) Kuinka ryhtyä moderaattoriksi? Sivun edistymistä ei ole arvioitu. Arvostuksen määrää ei ole arvioitu (ks. peer review).
Lisää dataa {{#opasnet_base_link:Op_fi2934}}

HUOM! Lue erityisesti sivu op_en:Object-oriented programming in Opasnet, joka on ensisijainen sivu tälle sisällölle.

Object-oriented programming is an approach where programs (or, in Opasnet, typically assessment models) have a modular structure in such a way that each part is considered as a separate object that has specific properties and interacts with other objects in standard ways.

Question

How should object-oriented programming be utilised in Opasnet in such a way that

it has seamless connections to R-tools,
it is easy to understand by non-expert users and contributors,
it uses the variable structure and other information structures (e.g. universal object) used in open assessment, and
it enables standards for typical processes in environmental health assessments (such as distribution modelling, life tables, decision optimising, etc.).

Answer

Structure of objects

Objects have two different implementations: wiki page in Opasnet, and S4 class object called ovariable (open assessment variable) in R-tools. The wiki page is the user-friendly interface for users, and ovariable is the versatile format for efficient, standardised modelling. The default direction for data is long (using the terminology in the merge function).

----#: . Should we have attribute "target" that defines the target of the variable estimate. For example, "height" may estimate the whole variation of heights of individuals in a population, or it may estimate the mean height of the population. Somehow the population, the target that is the basic unit (individual in this case) and the statistical parameter should be explicitly described. Can this be done by using vector attributes that have a value for each index column in the sample? Is this an index-specific issue, or variable-specific? --Jouni 07:50, 9 April 2012 (EEST) (type: truth; paradigms: science: comment)

Attribute	What it contains	How implemented in the wiki	How implemented in the R-tools as a S4 class object oavar
These attributes are needed in R-tools.
data	Observations, expert judgement, discussions, and other pieces of information.	Subheading under Rationale	Slot data = "data.frame". The data frame must contain Obs and Unit columns, at least one index column, and Result as the observation column. However, it must not contain Iter column.
sample	A random sample from the distribution (default is 10000 iterations).	Not shown	Slot sample = "data.frame". The data frame must contain columns Iter, Obs, Unit, one column for each index, and Result (if there are originally more than one observation columns, they are molten with melt function).
marginal	A Boolean vector with the size = number of indices = ncol(data) - 1. TRUE if an index is indexing a marginal distribution in sample, FALSE if joint distribution. The difference is that in a marginal distribution there are n iterations for each location of the index, while in joint distribution, there are altogether n iterations in such a way that the frequencies of locations match their probabilities.	Not implemented in wiki.	Slot marginal = "vector". Especially with indices with lots of locations, joint distribution needs much less memory.
formula	A computer code or algorithm to derive the answer from rationale and objects listed in dependencies. The formula may assume a deterministic dependency (e.g. y <- k*x + b), a conditional probability structure (y ~ dnorm(x, sd)), or a rank correlation matrix.	Subheading under Rationale, often using <rcode> tags.	Slot formula = "list". There may be several competing algorithms. Each of them is described (as a function?) as one entry in the list. When implementing the formula, the algorithm that is implemented is randomly selected for each iteration with equal probability unless otherwise specified in formula.prob.
These attributes are not (yet) implemented in R-tools.
question	A research question that defines the topic of the object	First main heading	Slot question = "character". Contains the question as text.
answer	The current best answer to the question, shown as text, data table, or distribution.	Second main heading; contains a single data table. NOTE! The data table is actually under ratonale/data but often it is the same as answer. The actual answer is precisely described by distribution and sample (see below).	Only sub-attributes are implemented.
locations	List of names for observation columns in the wide format (columns that are not indices), i.e. the same as measure.vars parameter in the function melt.	The same as locations parameter in t2b tag.	Slot locations = "vector". A vector with all observation column names. Can be integer (observation column position) or string (observation column name). By default, the column name for locations is "Parameter" and the column name for the actual observations is "Result". ⇤--#: . Do we actually need this in R, if always a variable is molten using melt before actual use? --Jouni 14:54, 4 April 2012 (EEST) (type: truth; paradigms: science: attack)
observation	An identifier of an individual when the answer consists of a group of individuals.	Obs column (usually implicit because the default is that each row is an observation) in the data table.	Obs column in data.frames data and sample. Not explicitly needed as a slot in S4 object. ----#: . How do we operate, if there are different Obs in different variables and they are merged? a) Repeat shorter Obs's until they reach the length of the longest. b) Refuse to operate unless the user renames or removes all but one Obs column; however if Obs only has 1 observation, it can be temporarily removed during merge. --Jouni 14:54, 4 April 2012 (EEST) (type: truth; paradigms: science: comment) ←--#: . I prefer b). a) is too abstract so that the user is just confused. --Jouni 14:54, 4 April 2012 (EEST) (type: truth; paradigms: science: defence)
iteration	An identifier of a probabilistic run or iteration. Sometimes it is also called a possible world or realisation.	Iter column in the data table (data usually not shown probabilistically in wiki).	Iter column in the data.frames sample (and rarely in data). Not explicitly needed as a slot in S4 object.
distribution	A joint probability distribution (with indices as dimensions) describing the answer mathematically.	Not shown	Slot distribution = "distribution?". A distribution created with e.g. dnorm(0,1). ----#: . We don't know yet how to actually implement this and how the indices are included. --Jouni 14:54, 4 April 2012 (EEST) (type: truth; paradigms: science: comment)
rationale	Any information that is needed to convince a critical reader that the answer is good.	Third main heading	Only sub-attributes are implemented.
unit	The measurement unit(s) that are used in the answer to measure the topic. The format used is kg m^2 /s^2 where a space implies a multiplication.	Subheading under Rationale with plain text. Also mentioned in the data table with parameter unit. If data table rows differ in units, there must be a Unit index.	There is no separate slot for unit. The unit is merged with data and subsequently with sample. This must be done, because if rows are ordered, it is impossible to attach units to right rows based on separate information.
dependencies	List of upstream objects that are causally related to this object.	Subheading under Rationale with a list of links to upstream (and sometimes downstream) wiki pages.	Slot dependencies = "vector". A character vector where entries have the format "Op_fi:Vaativuusluokkien keskipalkat". If the wiki identifier is omitted, the default is op_en.
formula.prob	A list of probabilities assigned to the competing algorithms in formula. The default is that each has an equal probability.	A detail in <rcode> code.	Slot formula.prob = "vector". Should have the same size as formula.

Methods

R code should be developed in such a way that there are object-specific implementations of critical functions. The user should see straightforward content, and all messy indexing etc should happen behind scenes.

These methods should be implemented for ovariable objects.

show, print: show the data slot.
plot: plot the sample, showing one (the first by default) marginal index with all locations and all other marginal indices with the first location only.
tidy: applies to data: remove id column; add Obs and Iter columns if they do not exist; Change the direction from wide to long.
createSample: create sample directly from data using interp.input.
GetSample, GetData: extract sample and data from the object, respectively.
Ops: applies to sample: merge two ovariables based on index columns, then perform the Ops operation to the result columns.
standardUnits: Based on units, transform the result column of data to SI units using Unit transformations table; then update unit.
demarginalize: turn one specified index from marginal to joint format. This function has parameter jointlimit: if the length(index_i) * n > jointlimit, then index_i is demarginalized. The default for jointlimit is 1000000. n is the length of Iter.

Oavariable functions available: make, update, plot, print, orbind, merge, callGeneric Ops

----#: . Function merge only uses indices that are of type "marginal" and not indices that are of type "joint". For a non-ovariable data.frame, by default all indices are marginals. --Jouni 14:58, 30 April 2012 (EEST) (type: truth; paradigms: science: comment)

+ Näytä koodi - Piilota koodi

library(OpasnetBaseUtils)
n <- 5

#################### Defines the S4 class "ovariable" which is the basic building block in open assessments.
setClass(
	"ovariable", 
	representation(
		sample = "data.frame", 
		data = "data.frame", 
		marginal = "vector", 
		formula = "function", 
		dependencies = "list"
	)
)

#################### interpret takes a vector and makes a data.frame out of it (to be used in e.g. make.ovariable).
### It also changes abbreviations into probability samples.
interpret <- function(data) {
	sample <- NULL
	if(is.vector(data)) {data <- data.frame(Result = data)}
	if("Iter" %in% colnames(data)) {
		out <- data}
	else {
		test <- !is.na(as.numeric(as.character(data$Result)))
	
		for(i in 1:nrow(data)) {
			if(test[i]) {
				sample <- c(sample, rep(as.numeric(as.character(data[i, "Result"])), n))
			} else {
				samplingguide <- as.numeric(strsplit(gsub(" ", "", data[i, "Result"]), "-")[[1]])
					if(is.na(samplingguide[1]) | is.na(samplingguide[2])) {
						sample <- c(sample, rep(data[i, "Result"], n))
					} else {
						sample <- c(sample, runif(n, samplingguide[1], samplingguide[2]))
					}
			}
		}
	
	
		out <- as.data.frame(array(1:(n*nrow(data)*(ncol(data)+1)), dim = c(n*nrow(data), ncol(data) + 1)))
		colnames(out) <- c("Iter", colnames(data))
	
		for(i in colnames(data)) {
			out[i] <- rep(data[, i], each = n)
		}
		out$Iter <- 1:n
		out$Result <- sample
		
	}
	return(out)
}


########### make.ovariable takes a vector or data.frame and makes an ovariable out of it.
make.ovariable <- function(
	data, 
	formula = function(dependencies){return(dependencies)}, 
	dependencies = list(x = NULL))
	{if(class(data) == "ovariable") {
		out <- data} 
	else {
		if(is.vector(data)) {data <- data.frame(Result = data)}
		sample <- interpret(data)
		out <- new("ovariable", 
			sample = sample, 
			data = data,
			marginal = ifelse(colnames(sample) %in% c("Result", "Unit"), FALSE, TRUE), 
			formula = formula, 
			dependencies = dependencies)
#		out <- update(out)
	}
	return(out)
}

########### update.ovariable updates the sample of an ovariable based on data and function.
setMethod(
	f = "update", 
	signature = "ovariable", 
	definition = function(object) {
		dat <- data.frame(Source = "Data", interpret(object@data))
		dep <- object@dependencies
		for(i in 1:length(dep)) {
			if(length(grep("Op_(en|fi)", dep[[i]])) > 0) {
				dep[[i]] <- op_baseGetData("opasnet_base", dep[[i]])}
			else {
				if(!is.numeric(dep[[i]])) {
					dep[[i]] <- get(dep[[i]])
				}
			}
		}
		form <- data.frame(Source = "Formula", make.ovariable(object@formula(dep))@sample)
		object@sample <- orbind(dat, form)@sample
		return(object)
	}
)

############ merge of ovariables merges the 'sample' slot by index columns except 'Unit'.
setMethod(f = "merge", 
	signature = signature(x = "ovariable", y = "ovariable"),
	definition = function(x, y) {
		test <- intersect(colnames(x@sample[x@marginal]), colnames(y@sample[y@marginal]))
		x@sample <- merge(x@sample, y@sample, by = test, all = TRUE)
		return(x)
	}
)

setMethod(f = "merge", 
	signature = signature(x = "ovariable", y = "numeric"),
	definition = function(x, y) {
		y <- make.ovariable(y)
		return(merge(x, y))
	}
)

setMethod(f = "merge", 
	signature = signature(x = "numeric", y = "ovariable"),
	definition = function(x, y) {
		x <- make.ovariable(x)
		return(merge(y, x))
	}
)

########## Arithmetic operations of ovariables: first they are merged by index columns,
### then the operation is performed for the Result.x and Result.y columns.
### If one of the expressions is numeric, it is first transformed to ovariable.
setMethod(
	f = "Ops", 
	signature = signature(e1 = "ovariable", e2 = "ovariable"), 
	definition = function(e1, e2) {
		out <- merge(e1, e2)@sample
		colnames(out) <- gsub(".x", "", colnames(out))
		out$Result <- callGeneric(out$Result, out$Result.y)
		if(!is.null(out$Unit.y)) {out$Unit <- paste(out$Unit, "|(", out$Unit.y, ")", sep= "")}
		e1@sample <- out[, !colnames(out) %in% c("Result.y", "Unit.y")]
		return(e1)
	}
)

setMethod(
	f = "Ops", 
	signature = signature(e1 = "ovariable", e2 = "numeric"), 
	definition = function(e1, e2) {
		e2 <- make.ovariable(e2)
		e1 <- callGeneric(e1, e2)
		return(e1)
	}
)

setMethod(
	f = "Ops", 
	signature = signature(e1 = "numeric", e2 = "ovariable"), 
	definition = function(e1, e2) {
		e1 <- make.ovariable(e1)
		e1 <- callGeneric(e2, e1)
		return(e1)
	}
)

orbind <- function(x, y) {
		x <- make.ovariable(x)
		y <- make.ovariable(y)
		cols <- setdiff(colnames(y@sample), colnames(x@sample))
		col <- as.data.frame(array("NA", dim = c(1, length(cols))))
		colnames(col) <- cols
		if("Unit" %in% cols) {col[, "Unit"] <- "?"}
		temp <- cbind(x@sample, col)

		cols <- setdiff(colnames(x@sample), colnames(y@sample))
		col <- as.data.frame(array("NA", dim = c(1, length(cols))))
		colnames(col) <- cols
		if("Unit" %in% cols) {col[, "Unit"] <- "?"}
		x@sample <- rbind(temp, cbind(y@sample, col))
		return(x)
	}

setMethod(
	f = "plot",
	signature = signature("ovariable"),
	definition = function(x) {
		x <- x@sample
		plot(x$Source, x$Result, ylab = x[x$Source == "Data", "Unit"][1])
	}
)


#################### Defines the S4 class "oassessment" which is the object type for open assessments.
setClass(
	"oassessment", 
	representation(
		names = "data.frame", 
		vars = "list",
		decisionmakers = "data.frame"
	)
)

setMethod(
	f = "print",
	signature = signature("oassessment"),
	definition = function(x) {
		cat("Names\n")
		print(xtable(x@names), type = 'html')
		cat("Decisionmakers\n")
		print(xtable(x@decisionmakers), type = 'html')
	}
)

setMethod(
	f = "print",
	signature = signature("ovariable"),
	definition = function(x) {
		cat("Sample\n")
		print(xtable(x@sample[x@sample$Iter %in% 1, ]), type = 'html')
		cat("Formula\n")
		print(x@formula)
		cat("Dependencies\n")
		print(x@dependencies)
	}
)

########### init.vars creates S4 variable and makes a list out of them. 
########### NOTE! You must include the formula code from each variable page, otherwise formulas and dependencies are not updated. 
########### Parameters:
## names: a data.frame that has the structure of oassessment@name (Columns: Name, Identifier, Direction, Result)
init.vars <- function(names) {

	vars <- list()
	for(x in 1:nrow(names)) { # Objects with names as aliases are created and filled with data from Opasnet Base.

		temp <- tidy(
			op_baseGetData("opasnet_base", names$Identifier[x]), 
			direction = names$Direction[x]
		)
		temp <- make.ovariable(temp)
		if(exists(paste("formula.", as.character(names$Identifier[x]), sep =""))) 
			{temp@formula <- get(paste("formula.", as.character(names$Identifier[x]), sep =""))}
		if(exists(paste("dependencies.", as.character(names$Identifier[x]), sep ="")))
			{temp@dependencies <- get(paste("dependencies.", as.character(names$Identifier[x]), sep =""))}
		vars[as.character(names$Result[x])] <- temp
	}
	return(vars)
}

Example code

+ Näytä koodi - Piilota koodi

library(xtable)	
a <- data.frame(a = c("a", "b", "c", "d"), Obs = 1, Unit = "ug /m^3", Result = c(1, "3 - 4", 5, "7 - 8") )
print(xtable(a), type = 'html')
print(xtable(tidy(op_baseGetData("opasnet_base", "Op_en5483"))), type = 'html')

###########   Defining variable d

d <- make.ovariable(
	data = a, 
	formula = function(dependencies) {
		x <- tidy(dependencies$x)
		x <- x[x$Observation == "Per.person", ]
		x <- make.ovariable(x)
		x*10 + dependencies$b
	}, 
	dependencies = list(x = "Op_en5483", b = 1)
)

############## Calculating with d

d <- update(d)
plot(d)
print(xtable(d@sample[d@sample$Iter < 4, ]), type = 'html')

Important rules:

Never use ".x" in the name of an ovariable.
The observation column always must have name "Result".
The iteration (aka. sample, run) column always must have name "Iter".
In the data slot, there is always column "Obs". It defines the observations that are from the same individual. Of omitted, each row is assumed to be from different individual.
Data slot must not have Iter column, but sample may or may not have. Typically it does have it.
Variable n is reserved for number of iterations.
Column name Source is reserved for the source of the result (either Data of Formula).
After using make.ovariable(), use update() to update the sample of the ovariable based on both Data and Formula. This is not done automatically, because often there are problems to run the formula part of the code.

op_fi:Olio-ohjelmointi Opasnetissä

About oassessment (open assessments)

They are collections of information that are used on assessment level.
They contain the following parts:
- Data frame about variables included. It has columns name (wiki page name), identifier (page identifier such as Op_fi2898), and alias (name that is short but descriptive in its context and that is used in rcode about the variable such as exposure, erf.radon etc.) Also, R object saved with object.put should be downloadable using this data. Should we also have hashtags available? How?
- a list of ovariables (variables themselves, not just names)
- If a variable is in the dependencies of a variable used in an assessment, the first variable is only included partly. In practice, if sample is available that is used, otherwise data. However, formula and dependencies are NOT used because they are outside the boundaries of the assessment. Otherwise the assessment may grow into an endless web.
There are several ways to treat dependencies.
- Upstream functions: deterministic functions that take upstream variables as inputs and calculate outputs. This is implemented first.
- JAGS or other hierarchical Bayesian approaches. What are the demands of this to oassessment? Should there be "model" or other slots for this?
- BBN. What are the demands of this to oassessment? At least conditional probabilities.
We need a wrapper to op_baseGetData in such a way that the code may give a wiki page name, an identifier, or an alias, and the function always knows what data to download. This makes the rcode more user-friendly. But does it always need a oassessment, or does it get the identifier based on wiki page name from the wiki database directly? The latter would be more useful.

There should be a function that automatically drops dimensions of an ovariable. This is the priority list:

First drop dimensions that are explicitly mentioned in parameter "drop" (a vector of index names).
Drop dimensions that cause the size of the ovariable to grow larger than dropsize = 1000000.
If there is enough information available to perform a VOI analysis, drop all dimensions except those VOIsize = 3 indices that have the largest VOI. (But VOI is attached to ovariables, not indices; how does this work out? It works out in the way that P(B) is thought as a variable and its VOI can be estimated. The VOI of P(A) or P(A|B) is calculated by giving the original ovariable as parameter.)
Never drop indices that are mentioned in parameter keep (a vector of index names).
"Drop" function changes an index from marginal to joint. In practice, the size changes from size(index)*size(Iter)*size(other indices combined) to size(Iter)*size(other indices combined). The variable can also be thought as a conditional probability P(A|B) which is then changed to joint by sampling from index B using its probability: P(A|B)*P(B) = P(A).
If P(B) is omitted it is assumed that all instances of B are equally likely. Typically, P(B) is assessment-specific, and therefore there is a slot for all P(B)s in oassessment. This slot is a data.frame with columns Variable, Index, Location, P, Description.
However, even if an index is changed from marginal to joint, the column that contains information about the location of the index on each iteration is still kept.

Rationale

References

Related files